Iustin Pop [Tue, 12 Apr 2011 14:04:43 +0000 (16:04 +0200)]
Prevent ssconf values from having non-string values
For whatever reason, my test cluster managed to acquire
shared_file_storage_dir with a None value, instead of empty
string. This is not flagged in masterd itself, but the node daemon
will fail in writing the value to disk, as it calls len() on the
received value.
Since this is a bad case, we should detect it as soon as possible (we
basically shouldn't be able to set it), but in the meantime we at
least prevent ssconf writes with such values.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Fri, 8 Apr 2011 10:29:08 +0000 (12:29 +0200)]
Add some tests for the auto_balance attribute
It tests node add/remove secondary, rather than cluster-level N+1
checks, but it's better than nothing.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>
Iustin Pop [Fri, 8 Apr 2011 08:40:16 +0000 (10:40 +0200)]
Node operations: take into account auto_balance
This patch changes the add to secondary/remove from secondary code to
not deduct/add the instance's memory if the instance is not
auto_balanced.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>
Iustin Pop [Thu, 7 Apr 2011 15:14:16 +0000 (17:14 +0200)]
Read/write auto_balance via Text
This also means _another_ change in the text format; we really should
move to json…
The unittests are also update for the new 9-column layout and
additionally a bit of improvement is done.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>
Iustin Pop [Thu, 7 Apr 2011 15:00:49 +0000 (17:00 +0200)]
Read auto_balance via Rapi
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>
Iustin Pop [Thu, 7 Apr 2011 14:49:38 +0000 (16:49 +0200)]
Read auto_balance via Luxi
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>
Iustin Pop [Thu, 7 Apr 2011 15:00:32 +0000 (17:00 +0200)]
Show the auto_balance flag in the instance listing
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>
Michael Hanselmann [Mon, 11 Apr 2011 14:36:49 +0000 (16:36 +0200)]
cli: Replace hardcoded strings with constants
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 7 Apr 2011 14:23:44 +0000 (16:23 +0200)]
Add a new attribute to Instance.Instance
This will mirror Ganeti's be/auto_balance one, which we need to use to
properly match N+1 computations.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Fri, 8 Apr 2011 09:14:51 +0000 (11:14 +0200)]
Some more changes to Makefile.am for htools
I duplicate the BINARY= rule in the ghc invocation in order to be able
to silence the if, which was confusing.
Additionally, a new target for running just the htools unit-tests is
provided.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Thu, 7 Apr 2011 10:42:09 +0000 (12:42 +0200)]
htools: Make opcode naming consistent with Ganeti codebase
This patch just cleans up the htools codebase to make it more consistent
with the naming of the Ganeti codebase.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 7 Apr 2011 10:03:51 +0000 (12:03 +0200)]
Merge branch 'devel-2.4'
* devel-2.4:
LUInstanceQueryData: Don't acquire locks unless requested
Increase the lock timeouts before we block-acquire
daemon.py: move startup log message before prep_fn
Display the actual memory values in N+1 failures
ssh.VerifyNodeHostname: remove the quiet flag
Add error checking and merging for cluster params
RAPI: Document need for Content-type header in requests
Fix output for “gnt-job info”
watcher: Fix misleading usage output
Clarify --force-join parameter message
locking: Fix race condition in lock monitor
utils: Export NiceSortKey function
Revert "Only merge nodes that are known to not be offline"
cluster-merge: only operate on online nodes
Only merge nodes that are known to not be offline
Treat empty oob_program param as default
Fix bug in instance listing with orphan instances
Fix bug related to log opening failures
Bump version for 2.4.1 release
cfgupgrade: Fix critical bug overwriting RAPI users file
Conflicts:
NEWS: Trivial
lib/opcodes.py: Added parameter descriptions, used variable for
"use_locking"
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Thu, 7 Apr 2011 09:44:52 +0000 (11:44 +0200)]
Merge branch 'stable-2.4' into devel-2.4
* stable-2.4:
Add error checking and merging for cluster params
Clarify --force-join parameter message
Treat empty oob_program param as default
Fix bug in instance listing with orphan instances
Fix bug related to log opening failures
Bump version for 2.4.1 release
cfgupgrade: Fix critical bug overwriting RAPI users file
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 7 Apr 2011 08:19:35 +0000 (10:19 +0200)]
OpCodes.hs: make allow_failover optional
And default to False, like in the Python codebase.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Wed, 6 Apr 2011 17:00:27 +0000 (19:00 +0200)]
htools: add an utility function for JSON parsing
This allows extracting values from a JSON object that might miss, but
have a well-defined default value.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Thu, 7 Apr 2011 07:45:43 +0000 (09:45 +0200)]
Two small Makefile fixes related to htools
First, fix hs-coverage on non-pristine tree, where the index.html file
already existed, and second, disallow compilation of htools binaries
if configure, for some reason, didn't enable them.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
René Nussbaumer [Wed, 6 Apr 2011 16:29:28 +0000 (18:29 +0200)]
htools: Use OpMigrateInstance with allow_failover option
Before hbal decided on the fly if an instance is migratable or not. As
we implemented failover fallback in commit
d5cafd31456 we can start to
use that.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 6 Apr 2011 16:32:31 +0000 (18:32 +0200)]
LUInstanceQueryData: Don't acquire locks unless requested
Until now LUInstanceQueryData always acquired locks for the instance(s)
and nodes involved. In combination with long-running operations this
prevented the use of “gnt-instance info”, even with the “--static”
option. With this patch, locks are only acquired when explicitely
requested in the opcode (like all query operations).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Tue, 29 Mar 2011 09:12:25 +0000 (11:12 +0200)]
gnt-instance migrate: Adding --allow-failover option
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Mon, 28 Mar 2011 12:54:33 +0000 (14:54 +0200)]
TLMigrateInstance: Merge failover code, allow fallback
As the code for failover for checking is almost identical it's an easy
task to switch it over to the TLMigrateInstance. This allows us to
fallback to failover if migrate fails prereq check for some reason.
Please note that everything from LUInstanceFailover.Exec is taken over
unchanged to TLMigrateInstance._ExecFailover, only with adaption to
opcode fields and variable referencing, but not in logic. There still
needs to go some effort into merging the logic with the migration (for
example DRBD handling). But this should happen in a separate iteration.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Mon, 4 Apr 2011 13:59:39 +0000 (15:59 +0200)]
Increase the lock timeouts before we block-acquire
This has been observed to cause problems on real clusters via the
following mechanism:
- a long job (e.g. a replace-disks) is keeping an exclusive lock on an
instance
- the watcher starts and submits its query instances opcode which
wants shared locks for all instances
- after about an hour, the watcher job falls back to blocking acquire,
after having acquired all other locks
- any instance opcode that wants an exclusive lock for an instance
cannot start until the watcher has finished, even though there's no
actual operation on that instance
In order to alleviate this problem, we simply increase the max timeout
until lock acquires are sent back to either blocking acquire or
priority increase. The timeout is computed such that we wait ~10 hours
(instead of one) for this to happen, which should be within the
maximum lifetime of a reasonable opcode on a healthy cluster. The
timeout also means that priority increases will happen every half hour.
We also increase the max wait interval to 15 seconds, otherwise we'd
have too many retries with the increased interval.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Wed, 30 Mar 2011 15:54:21 +0000 (17:54 +0200)]
utils: Add function generating regex for DNS name globbing
The intent of this function is to be able to provide a globbing operator
or query filters. One should be able to say, for example, something to
the effect of “gnt-instance shutdown '*.site'”.
Also rename a variable in MatchNameComponent.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 31 Mar 2011 16:43:25 +0000 (18:43 +0200)]
Verify file consistency using centrally computed list
Until now “gnt-cluster verify” (LUClusterVerify) would compute its own
list of files to check for consistency. This list was not complete and
certain inconsistencies were missed.
With this patch the code is changed to use the list of files used by
LUClusterRedistConf. The new check needs to be on a whole-cluster level,
and no longer per node.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 31 Mar 2011 16:39:52 +0000 (18:39 +0200)]
cmdlib: Factorize computation of ancillary files
… and change the logic in _RedistributeAncillaryFiles. The virtually
same list of files will be used to verify the files' consistency.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Fri, 1 Apr 2011 12:16:33 +0000 (14:16 +0200)]
qlang: Remove OP_GLOB operator
It'll be implemented using OP_REGEXP by the parser.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Wed, 30 Mar 2011 16:38:44 +0000 (18:38 +0200)]
query: Add implementation of regex match operator
So far this operator was not implemented. This patch adds an additional
value preparation function to the function table for binary operators,
used to compile the regular expression. Unittests are included.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Mon, 4 Apr 2011 16:21:53 +0000 (18:21 +0200)]
cmdlib: Fix mistake made in commit
75c7520f0
Commit
75c7520f0 used the wrong constant. I double-checked all other
changes made in the commit.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Mon, 4 Apr 2011 14:40:23 +0000 (16:40 +0200)]
cmdlib: Replace hardcoded values with constants
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Mon, 4 Apr 2011 10:13:44 +0000 (12:13 +0200)]
daemon.py: move startup log message before prep_fn
Before this, the output in the rapi daemon log was:
2011-04-04 03:09:51,026: ganeti-rapi pid=17447 INFO Reading users file
at /var/lib/ganeti/rapi/users
2011-04-04 03:09:51,027: ganeti-rapi pid=17447 INFO ganeti-rapi daemon
startup
Which is confusing, as it might look like the read of the users file
is part of the previous run. This is because we log the 'daemon
startup' message after the prepare_fn, which can log things on its
own.
The patch simply moves the 'daemon startup' message just before
prepare_fn call.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 4 Apr 2011 09:33:01 +0000 (11:33 +0200)]
Display the actual memory values in N+1 failures
This changes the display from:
Mon Apr 4 02:29:46 2011 * Verifying N+1 Memory redundancy
Mon Apr 4 02:29:46 2011 - ERROR: node node2: not enough memory to
accomodate instance failovers should node node1 fail
To:
Mon Apr 4 02:32:50 2011 * Verifying N+1 Memory redundancy
Mon Apr 4 02:32:50 2011 - ERROR: node node2: not enough memory to
accomodate instance failovers should node node1 fail (33536MiB needed,
27910MiB available)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Adeodato Simo [Wed, 30 Mar 2011 15:20:22 +0000 (16:20 +0100)]
Update iallocator.rst for multi-reloc mode
Signed-off-by: Adeodato Simo <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Fri, 1 Apr 2011 12:42:07 +0000 (14:42 +0200)]
RAPI: Convert instance shutdown to the new FillOpCode
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Thu, 31 Mar 2011 12:31:06 +0000 (14:31 +0200)]
Fix QA breakage caused by
3fd7f6524
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Thu, 31 Mar 2011 16:41:09 +0000 (18:41 +0200)]
ssh.VerifyNodeHostname: remove the quiet flag
This is not needed for this function, and can interfere with debugging
of ssh failures.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 31 Mar 2011 16:07:35 +0000 (18:07 +0200)]
Add a simple wrapper over utils.Retry
The new wrapper makes moving legacy code to utils.Retry or adding
retries in existing code simpler.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 31 Mar 2011 10:52:52 +0000 (12:52 +0200)]
Automatically enable hail if enabled and found
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 31 Mar 2011 10:49:07 +0000 (12:49 +0200)]
Expose whether htools was enabled to Python code
This exports whether htools was enabled at configure-time, and adds a
constant for our reference iallocator.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Thu, 31 Mar 2011 08:21:54 +0000 (10:21 +0200)]
test.ganeti.process_unittest: Fix race condition
There was a race condition on heavily loaded testsystem causing randomly
to fail the timeout unittests as the signal handler is not yet setup but
the timeout has already hit.
Therefore we introduce a workaround to wait until a program reached a
certain point (for example after signal handling setup) before we
actually go for the real run. The wait of course has a timeout as well,
but it's pretty high. If we hit the 20 seconds we have really big issues
anyway.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 31 Mar 2011 08:51:31 +0000 (10:51 +0200)]
Improve references to htools in the documentation
Was not sure about the bit in admin.rst, hope it's fine.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Wed, 30 Mar 2011 12:00:31 +0000 (14:00 +0200)]
Clarify the need for QuickCheck/Haskell tests
Expands the devnotes.rst doc and adds warnings in the Makefile.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Wed, 30 Mar 2011 11:32:26 +0000 (13:32 +0200)]
RAPI client: Remove support for version 0 instance creation requests
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 30 Mar 2011 10:52:02 +0000 (12:52 +0200)]
RAPI server: Drop support for instance creation format 0
Ganeti 2.1.3, released in June 2010, added support for a new, extensible
instance creation request format, called version 1. This patch removes
support for the old and undocumented version 0 format.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Simeon Miteff [Mon, 28 Mar 2011 20:47:36 +0000 (22:47 +0200)]
Improved GanetiRapiClient docstrings
- Added @rtype and/or @return where missing
- Fixed @param for Query() filter_ parameter (colon was missing)
Signed-off-by: Simeon Miteff <simeon.miteff@gmail.com>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Adeodato Simo [Fri, 25 Mar 2011 20:57:44 +0000 (20:57 +0000)]
Add design for inter-group instance moves (multi-reloc)
Signed-off-by: Adeodato Simo <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Adeodato Simo [Thu, 24 Mar 2011 21:28:47 +0000 (21:28 +0000)]
iallocator.rst: refactor for readability; minor improvements
This commits breaks down the "Input message" section of iallocator.rst into
two separate subsections: one detailing keys that are required in all
operation types; a second one detailing the "request" element, which is
different for each type of request.
Some other minor improvements are included as well:
- update input example to version 2, and add the "nodegroups" and
"enabled_hypervisors" top-level elements, and the "group" and
"hypervisor" attributes for nodes and allocation request, respectively.
- sort keys in the example dictionaries according to the order in earlier
sections, for easy comparison of documentation with its examples.
Signed-off-by: Adeodato Simo <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Stephen Shirley [Fri, 25 Feb 2011 15:01:38 +0000 (16:01 +0100)]
Add error checking and merging for cluster params
Set the default stderr logging level to WARNING so the relevant output
can be seen.
Signed-off-by: Stephen Shirley <diamond@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
René Nussbaumer [Mon, 28 Mar 2011 08:50:08 +0000 (10:50 +0200)]
Relax instance ERROR on admin_down on offline node
This fixes a issue, where an stopped instances is reported as ERROR
in cluster verify if it lives on a offline node. As the instances is
down this shouldn't happen.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Fri, 25 Mar 2011 13:22:24 +0000 (14:22 +0100)]
Implement submitting jobs from logical units
The design details can be seen in the design document
(doc/design-lu-generated-jobs.rst).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Adeodato Simo [Wed, 23 Mar 2011 16:49:24 +0000 (16:49 +0000)]
iallocator.rst: give pointers for alloc_policy semantics
Signed-off-by: Adeodato Simo <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Adeodato Simo [Wed, 23 Mar 2011 16:41:54 +0000 (16:41 +0000)]
Doc fix in iallocator.rst: multi-evac requires "evac_nodes"
The request argument for multi-evacuate mode is "evac_nodes", not "nodes"
(the example later in the file has the correct name already).
Signed-off-by: Adeodato Simo <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 24 Mar 2011 15:28:19 +0000 (16:28 +0100)]
watcher: improve logging a bit
Add some debug logging to detail why we don't run some steps.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Thu, 24 Mar 2011 14:13:12 +0000 (15:13 +0100)]
RAPI: Document need for Content-type header in requests
This was added to the NEWS file in commit
ab221ddf, but never
documented properly.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 24 Mar 2011 11:51:31 +0000 (12:51 +0100)]
Fix output for “gnt-job info”
If the result of an opcode was a non-empty dictionary, it
would be impossible to differenciate between input and result:
Input fields:
[…]
debug_level: 0
fields: cluster_name,master_node,volume_group_name
jobs: [[True, u'37922'], [True, u'37923'], [True, u'37924']]
Expected output:
Input fields:
[…]
debug_level: 0
fields: cluster_name,master_node,volume_group_name
Result:
jobs: [[True, u'37922'], [True, u'37923'], [True, u'37924']]
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Mon, 21 Mar 2011 15:47:47 +0000 (16:47 +0100)]
Remove old ensure-dirs (no longer needed)
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Fri, 18 Mar 2011 09:30:09 +0000 (10:30 +0100)]
Rewrite of ensure-dirs in python
I provided unittest to test the important pieces of the infrastructure.
The one remaining function (ResuriveEnsure) is not easy to unittest
but also not critical if it fails to operate correctly.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 23 Mar 2011 15:23:50 +0000 (16:23 +0100)]
hs-coverage: make a symlink hpc_index.html
This allows Apache to display the directory in a nicer way.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 23 Mar 2011 15:06:50 +0000 (16:06 +0100)]
Another attempt at fixing htools build without curl
OK, my previous small fix was not good. There is another issue: haddoc
(the documentation generator) needs to pass the same compiler options
(i.e. in our case, -DNO_CURL) to ghc. But in case of no curl, then it
shouldn't scan at all the RAPI library, as that is not used in our
builds.
Clearly, this is not a nice thing. So this patch changes from
including/excluding RAPI conditionally (in two places, the
ExtLoader.hs module and in hscan.hs), to always include RAPI, and
moves the curl/no curl login to RAPI itself, where it belongs.
Together with passing --optghc to haddock, this makes the builds
consistent both with and without RAPI. I also undo the removal of RAPI
from QC.hs.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 23 Mar 2011 14:42:09 +0000 (15:42 +0100)]
Fix some lint warnings in htools code
hlint gives more suggestions, but some make the code (IMHO) harder to
read.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 23 Mar 2011 12:25:22 +0000 (13:25 +0100)]
Fix lint errors in the htools code
These are just changes from hlint suggestions. Still compiles and
passes unittests.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Wed, 23 Mar 2011 16:01:13 +0000 (17:01 +0100)]
Add opcode summary to SubmitManyJobs errors
Requested-by: Iustin Pop <iustin@google.com>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 28 Feb 2011 17:04:45 +0000 (18:04 +0100)]
Add design for submitting jobs from LUs
This patch adds a design document describing how jobs can be submitted
from within LUs.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 23 Mar 2011 12:54:27 +0000 (13:54 +0100)]
Fix Haskell unittests without RAPI
Since we don't test for now the RAPI backend directly, we can simply
skip the import. Later we can make a conditional import if needed.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Tue, 1 Feb 2011 13:47:25 +0000 (14:47 +0100)]
Add import/export version 2 design document
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Jan 2011 18:43:25 +0000 (19:43 +0100)]
Add design document for lighttpd as HTTP server
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Jan 2011 18:42:45 +0000 (19:42 +0100)]
Add design document for X509 CA
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 23 Mar 2011 11:25:45 +0000 (12:25 +0100)]
Wrap long lines in configure.ac
- Use m4_normalize to make single-line strings while removing
unnecessary spaces
- Wrap lines longer than 80 characters
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Tue, 22 Mar 2011 17:58:55 +0000 (18:58 +0100)]
Update INSTALL and devnotes.rst with Haskell notes
This documents the needed libraries for Haskell development. It also
fixes a tiny typo in devnotes.rst.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 22 Mar 2011 17:18:36 +0000 (18:18 +0100)]
Revert and change the apidoc and coverage dirs
Based on Michael's suggestion, this patch partially reverts my
changes. The new directories are:
- doc/api/py
- doc/api/hs
- doc/coverage/py
- doc/coverage/hs
Basically the Python-specific output moves one level down (into py/)
compared to the original location, and the Haskell stuff goes into
hs/.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 22 Mar 2011 17:06:50 +0000 (18:06 +0100)]
Rename away htools/NEWS
Also add mention about it being obsolete.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 22 Mar 2011 17:04:52 +0000 (18:04 +0100)]
Cleanup the Haskell-related Makefile.am variables
This should be more readable now. I wanted to even use the nicer
_SOURCES, but _SOURCES is special in Automake (again), so _SRCS it is.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 22 Mar 2011 10:42:35 +0000 (11:42 +0100)]
Remove obsolete htools/Makefile
Only one target wasn't ported over (the TAGS one), as hasktags is not
available easily in distributions, so it doesn't make sense to enable
it for all developers.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 22 Mar 2011 10:35:22 +0000 (11:35 +0100)]
Move hlint rule to the main Makefile
Since we do have errors currently, this is not enabled from the main
'make lint' rule. That will get cleaned up later.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 22 Mar 2011 10:26:13 +0000 (11:26 +0100)]
Port the live-test htools rule to the main Makefile
This was a bit tricky, as the compilation from the top-dir changes the
paths in the .tix/.mix files.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 22 Mar 2011 09:47:59 +0000 (10:47 +0100)]
Enable htools apidoc generation and unify dir names
Previously, Python api doc was under doc/api (which didn't match with
the target rule, apidoc). After this patch, we have the following:
- make py-apidoc generates Python api doc under doc/py-apidoc
- make hs-apidoc generates Haskell api doc under doc/hs-apidoc
- make apidoc does both (if hs-apidoc enabled at configure time)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 21 Mar 2011 15:14:49 +0000 (16:14 +0100)]
Enable htools unittests and coverage reports
Since we have two sets of coverage data, we make the naming
consistent:
- doc/py-coverage for Python coverage
- doc/hs-coverage for Haskell coverage
We also need to exclude some more dirs from the list of checked
directories.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 21 Mar 2011 14:21:20 +0000 (15:21 +0100)]
Fix parallel compilation for htools
We do it via per-target-binary .o/.hi files. There are a few other
options, in the end this was chosen as different binaries can have
different options/components (e.g. hail won't use RAPI, etc.) and that
the unittests need different compilation options.
We add a wildcard *.o/*.hi to CLEANFILES, so that it removes all
per-target build artifacts.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 21 Mar 2011 12:51:21 +0000 (13:51 +0100)]
htools: add option to control the use of RAPI
RAPI needs the curl module, which is a binding to the libcurl C
library. This is more complex (can be compiled either against gnutls
or openssl, etc.) and it's not of much help when the tools are used on
the cluster itself, we disable it by default.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 21 Mar 2011 12:15:15 +0000 (13:15 +0100)]
htools: install hail in the iallocator dir
Instead of bindir. We do this by adding a new allocdir variable, and
redirecting hail to it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 21 Mar 2011 11:35:32 +0000 (12:35 +0100)]
Remove some obsolete htools files
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 21 Mar 2011 11:33:53 +0000 (12:33 +0100)]
Integrate htools's gitignore in top-level one
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 21 Mar 2011 11:25:07 +0000 (12:25 +0100)]
Makefile.am: add rules for htools compilation
This patch adds some (not very nice) rules for htools
compilation. Since automake only knows about some languages, and
doesn't have a real extension mechanism, I cheat and declare them as
"buildable scripts", instead of programs. Rationale:
automake support for dir_PROGRAMS is limited to a few languages; in
all cases, it assumes that translation from sources to binaries occurs
in two steps, compilation and linking; it's not easy to change its
assumption.
In our case, where “ghc --make” does both steps, it's hard to force it
into the automake model.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 21 Mar 2011 10:00:06 +0000 (11:00 +0100)]
configure.ac: add checks for Haskell compiler/libs
This patch adds an enable/disable option for htools (--enable-htools),
and associated tests for determining whether GHC (the compiler we use
for htools) and required libraries are present.
The method to do so is not very nice; usually, Haskell programs are
configured and compiled using cabal (http://www.haskell.org/cabal/, a
tool similar to Python's setuptools)), but that doesn't suit itself to
nice integration with autoconf/automake, so we test for the modules
presence manually.
In the end, we set a few variables:
- GHC: the path to the ghc compiler
- HTOOLS_MODULES: command line option for ghc to select the wanted
'parallel' module
- HTOOLS_NOCURL: set to -DNO_CURL if we don't want to enable curl (and
thus RAPI) support in htools
- HTOOLS: set to 'yes' if we should compile/install the htools
programs
- HTOOLS_APIDOC: set to yes if we should build/install the htools
API documentation
- WANT_HTOOLS, WANT_HTOOLSAPIDOC: two automake conditionals for later
use in Makefile.am
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Fri, 18 Mar 2011 14:08:30 +0000 (15:08 +0100)]
RAPI client: Tidy and test WaitForJobCompletion
- Use constants
- Don't sleep if no delay is given
- Mark function as deprecated: it uses polling instead of waiting for changes
(but the latter needs authentication); it can still be used
- Add unittests
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Fri, 18 Mar 2011 14:05:43 +0000 (15:05 +0100)]
RAPI client: Add job status constants
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Fri, 18 Mar 2011 12:50:32 +0000 (13:50 +0100)]
RAPI client: Job IDs are strings
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Fri, 18 Mar 2011 12:08:08 +0000 (13:08 +0100)]
Split BuildHooksEnv of LUs
Commit
dd7f677623 added another call to BuildHooksEnv to provide
post-phase status variables. Since BuildHooksEnv also built the node
lists, that meant they have to be built twice. First a rather strict
check was used, but it turned out to be more tricky. Commit
b423c51336
had to remove the strict check again.
With this patch the function is split in two parts, one generating the
actual environment variables, and another part returning the node lists.
The former is called twice.
Unittests are updated.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Apollon Oikonomopoulos [Fri, 18 Mar 2011 14:38:40 +0000 (16:38 +0200)]
cfgupgrade: really ignore hostname when told so
cfgupgrade provides a --ignore-hostname option, however consequently it
spawns config.ConfigWriter, which refuses to handle a foreign
configuration file by default.
config.ConfigWriter provides a relative keyword argument,
accept_foreign, so we tie this to the value of options.ignore_hostname.
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Fri, 18 Mar 2011 10:22:39 +0000 (11:22 +0100)]
RAPI client: fix epydoc formatting
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Theo Van Dinter [Fri, 18 Mar 2011 09:39:18 +0000 (10:39 +0100)]
Add a helper function to the RAPI client
This adds a new method WaitForJobCompletion that can be used for
client who are not interested in the entire job log, just in its
completion status.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Thu, 17 Mar 2011 17:24:35 +0000 (18:24 +0100)]
Move htools-ganeti 2.3 design doc to doc/
I didn't know where to put in the TOC, so I added it after the ganeti
2.3 design doc.
Also, it seems that Sphinx is a bit limited in its list nesting
(compared to Pandoc), so I had to rework a bit the indentation (it
still doesn't generate good output, but…).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 17 Mar 2011 17:06:03 +0000 (18:06 +0100)]
Integrate htools man pages in the main tree
This moves the man pages to their right place, removes the static
footer and header, and adds Makefile rules for the new .1 man pages.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Thu, 17 Mar 2011 12:23:03 +0000 (13:23 +0100)]
Remove restrictive hook node list check
Commit
dd7f67762 added a restrictive check for the node lists returned
by BuildHooksEnv, leading to errors with some LUs, one of which was
fixed in commit
0dfa2c227. As it turns out, other LUs have similar
issues, some not easy to fix. This patch disables the restrictive check
until the BuildHooksEnv functions can be split into one part generating
the actual environment and one generating the node lists.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 17 Mar 2011 16:36:57 +0000 (17:36 +0100)]
watcher: Fix misleading usage output
When “ganeti-watcher” is called with an argument, it would hint at
a non-existing “-f” parameter. With this patch the separate usage
string is no longer necessary.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 17 Mar 2011 16:24:14 +0000 (17:24 +0100)]
Revert "Add temporary dir/file for htools import"
This reverts commit
8bc9e04a21457c4188475dc849d0d7d1e22ee929, not
needed anymore.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 17 Mar 2011 15:47:06 +0000 (16:47 +0100)]
Merge remote branch 'htools/master'
* htools/master: (605 commits)
Update NEWS file for version 0.3.1
Update copyright years in the man pages
Remove obsolete env vars from the manpages
Fix dist archive generation
luxi backend: show attribute names in errors
Update exit code information in the hbal man page
Fix broken hscolour.css symlink in tarball
Update NEWS file for the 0.3.0 release
Remove an obsolete TODO
Update live-test.sh w.r.t. simulation mode changes
Prevent allocation on unallocable node groups
Fix bug in simulation backend
Precompute allocation nodes
Change the AllocNodes type and functions
Extract node pair generation from tryAlloc
tryAlloc: restrict valid node pairs to same-group
Cluster.hs: add a new type alias
Rapi: read and use the vm_capable node flag
IAllocator: read and use the vm_capable node flag
IAllocator: replace fake policy with real one
...
Note: done via the following sequence of commands:
git checkout master && git pull
git remote add htools git://git.ganeti.org/htools.git
git remote update
git merge -s recursive -X subtree=htools htools/master
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 17 Mar 2011 10:06:20 +0000 (11:06 +0100)]
Add temporary dir/file for htools import
In order for “git merge -s recursive -X subtree=htools” to do its
magic, an (mostly empty) directory must exist in the target tree name
‘htools’. Hence we add a temp file to it before the merge, and remove
it after that.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Thu, 17 Mar 2011 10:52:23 +0000 (11:52 +0100)]
Fix hook node list when adding node
This broke QA (and everyone trying to add a node) by complaining about
different node lists.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Stephen Shirley [Thu, 17 Mar 2011 10:05:36 +0000 (11:05 +0100)]
Clarify --force-join parameter message
This isn't only used during cluster merge.
Signed-off-by: Stephen Shirley <diamond@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Wed, 16 Mar 2011 17:35:18 +0000 (18:35 +0100)]
hooks: Provide variables with post-opcode values
When a hook is called, it is provided with a number of variables
describing the status of the instance/node/etc. before the operation.
Some opcodes provide extra variables to see modified values from hooks,
but that's not a generic solution.
This patch modifies the code calling hooks to generate the environment
once before and once after an opcode has been executed. Doing so should
be safe—I did not find any LU.BuildHooksEnv modifying LU instance
attributes. The values collected after running the opcode are prefixed
with “GANETI_POST_”, as opposed to “GANETI_” for pre-execution
variables. The latter are still provided for backwards compatibility.
Environment variable examples:
gnt-instance start $instance:
GANETI_INSTANCE_STATUS=down
GANETI_POST_INSTANCE_STATUS=up
gnt-instance modify -B memory=512 $instance:
GANETI_INSTANCE_BE_memory=768
GANETI_POST_INSTANCE_BE_memory=512
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 16 Mar 2011 12:59:32 +0000 (13:59 +0100)]
HooksMaster: Add more assertions for variable names
Also replace explicit loop with dict.update.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>