Michael Hanselmann [Tue, 29 Nov 2011 16:32:48 +0000 (17:32 +0100)]
Change “node_info” RPC to accept multiple VGs/hypervisors
Keeping the node state up to date will require information from multiple
VGs and hypervisors. Instead of requiring multiple calls this change
allows a single call to return all needed information. Existing users
are changed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 29 Nov 2011 15:36:26 +0000 (16:36 +0100)]
locking: Allow checking if lock is owned in certain mode
With this patch the “LockSet” and “GanetiLockManager” classes have a new
function to check if a single or a group of locks (at a certain level)
have been acquired in a specific mode. This will be used for additional
assertions. Until now they could only check if a lock has been acquired,
but not in which mode. One use-case will be updating the node state in
various places, where the node lock must be acquired in exclusive mode.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 24 Nov 2011 13:59:52 +0000 (14:59 +0100)]
Merge branch 'devel-2.5'
* devel-2.5:
ConfigWriter: Fix epydoc error
ConfigWriter: Fix epydoc error
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 24 Nov 2011 12:22:28 +0000 (13:22 +0100)]
Merge branch 'devel-2.4' into devel-2.5
* devel-2.4:
ConfigWriter: Fix epydoc error
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Thu, 24 Nov 2011 12:15:46 +0000 (13:15 +0100)]
Merge branch 'stable-2.5' into devel-2.5
* stable-2.5:
ConfigWriter: Fix epydoc error
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Andrea Spadaccini <spadaccio@google.com>
Michael Hanselmann [Thu, 24 Nov 2011 12:02:36 +0000 (13:02 +0100)]
ConfigWriter: Fix epydoc error
The parameter is called “mods”, not “modes”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Andrea Spadaccini <spadaccio@google.com>
(cherry picked from commit
1730d4a1ab56ef36d082b614d3d0ab13f3e14a85)
Michael Hanselmann [Thu, 24 Nov 2011 12:02:36 +0000 (13:02 +0100)]
ConfigWriter: Fix epydoc error
The parameter is called “mods”, not “modes”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Andrea Spadaccini <spadaccio@google.com>
Michael Hanselmann [Thu, 24 Nov 2011 09:50:35 +0000 (10:50 +0100)]
Merge branch 'devel-2.5'
* devel-2.5:
LUGroupAssignNodes: Fix node membership corruption
LUGroupAssignNodes: Fix node membership corruption
Fix pylint warning on unreachable code
LUNodeEvacuate: Disallow migrating all instances at once
Separate OpNodeEvacuate.mode from iallocator
LUNodeEvacuate: Locking fixes
Fix error when removing node
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Thu, 24 Nov 2011 09:06:39 +0000 (10:06 +0100)]
Merge branch 'devel-2.4' into devel-2.5
* devel-2.4:
LUGroupAssignNodes: Fix node membership corruption
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Thu, 24 Nov 2011 08:40:10 +0000 (09:40 +0100)]
Merge branch 'stable-2.5' into devel-2.5
* stable-2.5:
LUGroupAssignNodes: Fix node membership corruption
Fix pylint warning on unreachable code
LUNodeEvacuate: Disallow migrating all instances at once
LUNodeEvacuate: Locking fixes
Fix error when removing node
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Thu, 24 Nov 2011 07:43:04 +0000 (08:43 +0100)]
LUGroupAssignNodes: Fix node membership corruption
Note: This bug only manifests itself in Ganeti 2.5, but since the
problematic code also exists in 2.4, I decided to fix it there.
If a node was assigned to a new group using “gnt-group assign-nodes” the
node object's group would be changed, but not the duplicate member list
in the group object. The latter is an optimization to require fewer
locks for other operations. The per-group member list is only kept in
memory and not written to disk.
Ganeti 2.5 starts to make use of the data kept in the per-group member
list and consequently fails when it is out of date. The following
commands can be used to reproduce the issue in 2.5 (in 2.4 the issue was
confirmed using additional logging):
$ gnt-group add foo
$ gnt-group assign-nodes foo $(gnt-node list --no-header -o name)
$ gnt-cluster verify # Fails with KeyError
This patch moves the code modifying node and group objects into
“config.ConfigWriter” to do the complete operation under the config
lock, and also to avoid making use of side-effects of modifying objects
without calling “ConfigWriter.Update”. A unittest is included.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(cherry picked from commit
218f4c3de706aca7e4521d7e1975f517cf5ecb9b)
Michael Hanselmann [Thu, 24 Nov 2011 07:43:04 +0000 (08:43 +0100)]
LUGroupAssignNodes: Fix node membership corruption
Note: This bug only manifests itself in Ganeti 2.5, but since the
problematic code also exists in 2.4, I decided to fix it there.
If a node was assigned to a new group using “gnt-group assign-nodes” the
node object's group would be changed, but not the duplicate member list
in the group object. The latter is an optimization to require fewer
locks for other operations. The per-group member list is only kept in
memory and not written to disk.
Ganeti 2.5 starts to make use of the data kept in the per-group member
list and consequently fails when it is out of date. The following
commands can be used to reproduce the issue in 2.5 (in 2.4 the issue was
confirmed using additional logging):
$ gnt-group add foo
$ gnt-group assign-nodes foo $(gnt-node list --no-header -o name)
$ gnt-cluster verify # Fails with KeyError
This patch moves the code modifying node and group objects into
“config.ConfigWriter” to do the complete operation under the config
lock, and also to avoid making use of side-effects of modifying objects
without calling “ConfigWriter.Update”. A unittest is included.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 24 Nov 2011 07:58:56 +0000 (08:58 +0100)]
Fix pylint warning on unreachable code
Commit
c50452c3186 added an exception when all instances should be
evacuated off a node, but did so in a way which made pylint complain
about unreachable code.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 23 Nov 2011 13:01:23 +0000 (14:01 +0100)]
LUNodeEvacuate: Disallow migrating all instances at once
There is a design issue in the iallocator interface which prevents us
from doing this.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Wed, 23 Nov 2011 12:40:46 +0000 (13:40 +0100)]
Separate OpNodeEvacuate.mode from iallocator
Until now the iallocator constants for node evacuation
(IALLOCATOR_NEVAC_*) were also used for the opcode. However, it turned
out this was due to a misunderstanding and is incorrect. This patch adds
new constants (with the same values) and changes the affected places.
Fortunately the RAPI client already used good names, so no changes are
necessary.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Wed, 23 Nov 2011 12:16:14 +0000 (13:16 +0100)]
LUNodeEvacuate: Locking fixes
When evacuating a node, only an assertion without informative text was
used to check if the necessary node locks had been acquired. This was on
top of evaluating the list of nodes without having a node group lock, so
this was changed as well.
Also update some exception messages to include “retry the operation”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 23 Nov 2011 07:15:18 +0000 (08:15 +0100)]
Fix error when removing node
ConfigWriter.GetAllInstancesInfo returns a dictionary, not a list.
Removing a node would fail with “too many values to unpack”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Guido Trotter [Mon, 21 Nov 2011 11:49:21 +0000 (11:49 +0000)]
manpages: update beparams explanations
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Tue, 22 Nov 2011 10:07:45 +0000 (10:07 +0000)]
constants: reindent a few dicts
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 21 Nov 2011 11:18:22 +0000 (11:18 +0000)]
Remove BE_MEMORY from beparams but keep compatibility
Queries are already compatible (be/memory is an alias for be/maxmem) and
import/exports work. This patch patch fixes it for cluster init, modify
and instance add/start/modify.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 21 Nov 2011 11:12:00 +0000 (11:12 +0000)]
burnin: use mem_size as max and min
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 21 Nov 2011 11:05:31 +0000 (11:05 +0000)]
unittests: use max/min memory
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 21 Nov 2011 10:56:50 +0000 (10:56 +0000)]
cmdlib: use MAXMEM for all operations
Since for now we can only start instances at their maximum memory, we
modify all checks to use that value. When we'll have better support for
using a value in between some of these checks have to move to minimum
memory.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 21 Nov 2011 10:43:28 +0000 (10:43 +0000)]
qa: use maximum and minimum memory
test modification of either parameter, but also both at once.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 21 Nov 2011 10:22:35 +0000 (10:22 +0000)]
hypervisors: use maximum memory for all operations
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Thu, 17 Nov 2011 14:09:32 +0000 (14:09 +0000)]
ImportExport: use max and min memory params
Import uses the old "memory" parameter to populate the two new ones, if
they're not overridden already.
FinalizeExport exports minmem and maxmem, but also memory, as maxmem, to
allow importing to older ganeti clusters.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Thu, 17 Nov 2011 14:39:43 +0000 (14:39 +0000)]
Query: allow query on maximum and minimum memory
be/memory is kept as an alias.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Thu, 17 Nov 2011 14:12:44 +0000 (14:12 +0000)]
ShowInstanceConfig: show max and min memory
The old "memory" value is kept as maxmem, for now, for
retrocompatibility.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Thu, 17 Nov 2011 15:07:55 +0000 (15:07 +0000)]
instance hooks: pass maximum and minimum memory
Also pass the "memory" value for retrocompatibility, for now.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Thu, 10 Nov 2011 16:19:12 +0000 (16:19 +0000)]
beparams: add min/max memory values
For now the new "memory" parameter stays there, but it will be removed
later. The new values are just taken from the old one, in this patch.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Andrea Spadaccini [Mon, 21 Nov 2011 10:43:28 +0000 (10:43 +0000)]
design-resource-model: update disk params section
Simplify design by moving all the parameters to disk template level,
explaining why this is sub-optimal. Add notes about DRBD versions,
corner cases and parameters application time.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Andrea Spadaccini [Wed, 16 Nov 2011 11:31:18 +0000 (11:31 +0000)]
Set DRBD sync speed in DRBD8.Assemble
Instead of relying on clients of the class for setting the device speed
(and, in general, the DRBD parameters), move this responsibility inside
the Assemble method.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 21 Nov 2011 09:34:53 +0000 (10:34 +0100)]
build-rpc: Fail if call is defined more than once
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Andrea Spadaccini [Fri, 18 Nov 2011 12:07:27 +0000 (12:07 +0000)]
Reapply commit 2a6de57 after merge
In the last merge I erroneously discarded the changes introduced by
commit 2a6de57 "Check the results of master IP RPCs". This commit
reintroduces them.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Mon, 21 Nov 2011 07:22:24 +0000 (08:22 +0100)]
Fix QA breakage caused by merge
0e82dcf9
Patch tested and confirmed to work by Andrea Spadaccini
<spadaccio@google.com>.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Andrea Spadaccini <spadaccio@google.com>
Michael Hanselmann [Thu, 17 Nov 2011 11:08:32 +0000 (12:08 +0100)]
masterd: Initialize job queue only after RPC client
Otherwise jobs started after an unclean master shutdown will fail as
they depend on the RPC client.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 17 Nov 2011 11:07:57 +0000 (12:07 +0100)]
masterd: Shutdown only once running jobs have been processed
Until now, if masterd received a fatal signal, it would start shutting
down immediately. In the meantime it would hang while jobs are still
processed. Clients couldn't connect anymore to retrieve a jobs' status.
This this patch masterd checks if any job is running before shutting
down. If there is it'll check again every five seconds. Once all jobs
are finished, it waits another five seconds to give clients a chance to
retrieve the jobs' status. After that masterd will shutdown in a clean
fashion.
If a second signal is received the old behaviour is preserved.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 17 Nov 2011 11:01:33 +0000 (12:01 +0100)]
daemon: Support clean daemon shutdown
Instead of aborting the main loop as soon as a fatal signal (SIGTERM or
SIGINT) is received, additional logic allows waiting for tasks to finish
while I/O is still being processed.
If no callback function is provided the old behaviour--shutting down
on the first signal--is preserved.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 17 Nov 2011 10:56:34 +0000 (11:56 +0100)]
daemon: Allow custom maximum timeout for scheduler
This is needed in case the scheduler user (daemon.Mainloop in this case)
has other timeouts at the same time. Needed for clean master shutdown.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 17 Nov 2011 10:55:18 +0000 (11:55 +0100)]
jqueue: Add code to prepare for queue shutdown
Doing so will prevent job submissions (similar to a drained queue),
but won't affect currently running jobs. No further jobs will be
executed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 16 Nov 2011 11:35:07 +0000 (12:35 +0100)]
workerpool: Export function to check for running tasks
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 16 Nov 2011 11:34:50 +0000 (12:34 +0100)]
daemon: Use counter instead of boolean for mainloop abortion
Also log a message when a fatal signal was received and use dict.items.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 17 Nov 2011 14:40:18 +0000 (15:40 +0100)]
htools: adjust imports for newer compilers
While testing with ghc 7.2, I saw that some imports we are using are
very old (from ghc 6.8 time), even though current libraries are using
different names.
We fix this and bump minimum documented version to ghc 6.12, as I
don't have 6.10 to test anymore (possibly still works with that
version, but better safe - both Ubuntu Lucid and Debian Squeeze ship
with 6.12 nowadays).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Andrea Spadaccini [Fri, 18 Nov 2011 11:27:16 +0000 (11:27 +0000)]
Merge branch 'devel-2.5'
* devel-2.5: (24 commits)
LUInstanceCreate: Release unused node locks
htools: rework message display construction
hbal: handle empty node groups
Document OpNodeMigrate's result for RAPI
Ensure unused ports return to the free port pool
Re-wrap a paragraph to eliminate a sphinx warning
Fix newer pylint's E0611 error in compat.py
Fail if node/group evacuation can't evacuate instances
Update init script description
LUInstanceRename: Compare name with name
LUClusterRepairDiskSizes: Acquire instance locks in exclusive mode
Update synopsis for “gnt-cluster repair-disk-sizes”
Move hooks PATH environment variable to constants
Check the results of master IP RPCs
Add documentation for the master IP hooks
Add master IP turnup and turndown hooks
Add RunLocalHooks decorator
Generalize HooksMaster
Update NEWS for 2.5.0~rc4
Bump version to 2.5.0~rc4
...
Conflicts:
NEWS
doc/hooks.rst
lib/backend.py
lib/cmdlib.py
lib/constants.py
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 18 Nov 2011 10:09:33 +0000 (11:09 +0100)]
Merge branch 'stable-2.5' into devel-2.5
* stable-2.5:
htools: rework message display construction
hbal: handle empty node groups
Document OpNodeMigrate's result for RAPI
Fail if node/group evacuation can't evacuate instances
LUInstanceRename: Compare name with name
LUClusterRepairDiskSizes: Acquire instance locks in exclusive mode
Update NEWS for 2.5.0~rc4
Bump version to 2.5.0~rc4
jqueue: Allow zero jobs to be submitted at once
hail: don't select the primary as new secondary
hail: add an extra safety check in relocate
Bump version to 2.5.0~rc3
Conflicts:
configure.ac: Trivial
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 18 Nov 2011 07:27:30 +0000 (08:27 +0100)]
Merge branch 'devel-2.4' into devel-2.5
* devel-2.4:
Ensure unused ports return to the free port pool
Re-wrap a paragraph to eliminate a sphinx warning
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Agata Murawska [Thu, 17 Nov 2011 14:59:39 +0000 (15:59 +0100)]
admin.rst update regarding offline state of the instance
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Wed, 16 Nov 2011 15:59:37 +0000 (16:59 +0100)]
NEWS update - offline instance state
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Wed, 16 Nov 2011 16:08:49 +0000 (17:08 +0100)]
Backwards compatibity - added admin_up to query
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Agata Murawska [Wed, 16 Nov 2011 15:41:15 +0000 (16:41 +0100)]
Man page update: online/offline state of instance
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 17 Nov 2011 11:33:28 +0000 (12:33 +0100)]
Add small node in admin.rst about confd disabling
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 17 Nov 2011 11:31:10 +0000 (12:31 +0100)]
Warn if we enable maintain-node-health without confd
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 17 Nov 2011 11:19:22 +0000 (12:19 +0100)]
Adapt daemon-util to ENABLE_CONFD
We still allow explicit shutdown of confd, but we prevent manual
or automatic start-up.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 17 Nov 2011 11:04:58 +0000 (12:04 +0100)]
Adapt watcher for ENABLE_CONFD
If confd is disabled, do not automatically restart it. Furthermore, we
can't run maintenance actions if it is disabled so log a warning.
Note that I haven't completely disabled the NodeMaintenance class with
ENABLE_CONFD = False because I think they are at two different levels
(e.g. we might have other maintenance actions done even with confd
disabled).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 17 Nov 2011 10:55:03 +0000 (11:55 +0100)]
Prevent runnning of confd tests in burnin
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 17 Nov 2011 10:49:56 +0000 (11:49 +0100)]
Add toggle for enabling/disabling confd
Doesn't do anything yet.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 17 Nov 2011 10:19:41 +0000 (11:19 +0100)]
Fix unittest bug related to offline instances
Currently, the code in Node.hs is overly strict: once a node's free
memory reaches 0, it will refuse to add any instances (offline or
not). I think this is a safe safeguard (I don't expect nodes to run
without at least 1MB of free memory), so rather than change this
behaviour we need to restrict the Node generation in the unittest to
skip such nodes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>
Iustin Pop [Wed, 16 Nov 2011 18:14:48 +0000 (19:14 +0100)]
htools: reindent the rest of the files
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 16 Nov 2011 17:53:47 +0000 (18:53 +0100)]
htools: re-indent IAlloc.hs
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 16 Nov 2011 17:19:42 +0000 (18:19 +0100)]
htools: reindent hspace
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 16 Nov 2011 17:18:16 +0000 (18:18 +0100)]
htools: reindent hbal
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 16 Nov 2011 17:15:57 +0000 (18:15 +0100)]
htools: reindent CLI.hs
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 16 Nov 2011 17:11:41 +0000 (18:11 +0100)]
htools: re-indent QC.hs
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 16 Nov 2011 16:54:09 +0000 (17:54 +0100)]
htools: re-indent Node.hs
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 16 Nov 2011 16:45:11 +0000 (17:45 +0100)]
htools: finish re-indenting Cluster.hs
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Wed, 16 Nov 2011 09:35:06 +0000 (10:35 +0100)]
masterd: Don't pass mainloop to server class
It is not used.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 16 Nov 2011 08:37:49 +0000 (09:37 +0100)]
workerpool: Allow processing of new tasks to be stopped
This is different from “Quiesce” in the sense that this function just
changes an internal flag and doesn't wait for the queue to be empty.
Tasks already being processed continue normally, but no new tasks will
be started. New tasks can still be added, but won't be processed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 16 Nov 2011 08:11:59 +0000 (09:11 +0100)]
workerpool: Use loop to ignore spurious notifications
This saves us from returning to the worker code when there is no
task to be processed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 16 Nov 2011 08:06:23 +0000 (09:06 +0100)]
jqueue: Factorize code checking for drained queue
This is in preparation for a clean(er) shutdown of masterd.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 17 Nov 2011 11:36:18 +0000 (12:36 +0100)]
LUInstanceCreate: Release unused node locks
After iallocator ran we can release any unused node locks. Since they
must be in exclusive mode this should improve parallelization during
instance creation.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 16 Nov 2011 07:08:34 +0000 (08:08 +0100)]
cmdlib.TLReplaceDisks: Use itertools.count
… instead of a variable which needs to be incremented for every step.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Tue, 15 Nov 2011 17:16:36 +0000 (18:16 +0100)]
htools: rework message display construction
While diagnosing some (unrelated) memory usage in htools, I've
stumbled upon some very bad behaviour in checkData: mapAccum is
non-strict, and the tuple we use also, so that results in the list of
list of messages being very bad space-wise (hundreds of MB of memory
for a simulated cluster with thousands of nodes, all with errors).
The new, explicit reuse of the old message list has a linear memory
behaviour. The only downside is that messages are listed in the
reverse order (which I'll fix on master).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 15 Nov 2011 10:15:34 +0000 (11:15 +0100)]
hbal: handle empty node groups
This patch changes an internal assert (which can only be triggered
when a node group is empty) into properly handling this case (and
returning empty node/instance lists).
While we could handle this in the backend (Cluster.splitNodeGroup)
this would actually mean than we change the behaviour for a cluster
with just two node groups, once of which is empty (where today we
don't require a node group argument).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 14 Nov 2011 14:33:20 +0000 (15:33 +0100)]
Start converting the indentation to 2-chars
… also do some other small style cleanups.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Agata Murawska [Tue, 15 Nov 2011 15:48:23 +0000 (16:48 +0100)]
htools: Nicer arbitrary instantiation
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Tue, 15 Nov 2011 10:28:40 +0000 (11:28 +0100)]
htools: change behaviour for offline instances
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Mon, 14 Nov 2011 13:54:32 +0000 (14:54 +0100)]
htools: remove 'running' from instance options
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Thu, 10 Nov 2011 10:21:32 +0000 (11:21 +0100)]
htools: change runSt type to InstanceStatus
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Thu, 10 Nov 2011 10:10:13 +0000 (11:10 +0100)]
htools: create InstanceStatus ADT
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Tue, 8 Nov 2011 10:05:43 +0000 (11:05 +0100)]
Transition into and out of offline instance state
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Tue, 8 Nov 2011 10:04:34 +0000 (11:04 +0100)]
Introduce admin_state as 3-values data type
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Thu, 27 Oct 2011 11:29:41 +0000 (13:29 +0200)]
Rename admin_up to admin_state
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Fri, 28 Oct 2011 11:22:02 +0000 (13:22 +0200)]
Fixed typo in _VerifyResultRow
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Tue, 15 Nov 2011 14:09:59 +0000 (15:09 +0100)]
algo: Make a dict from an flat list
This is in preparation to take deeper dict constructs from the command
line. You can feed the optionslist directly constructed of type
"identkeyval" to it and it returns a fully deflated dict.
This is mainly needed for the resource model changes where we have to
modify the disk_state which is a 3 level dict:
disk_type/name:disk_reserved=10g
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Tue, 15 Nov 2011 13:46:21 +0000 (14:46 +0100)]
locking: Make some aliased methods public
Some methods, such as “_is_owned” and “list_owned”, have been aliased to
make them public for a while now. This patch makes the actual
implementation public.
SharedLock's “is_owned” needs to be aliased to “_is_owned” to remain
compatible with Python's built-in threaded.Condition class.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 15 Nov 2011 11:23:43 +0000 (12:23 +0100)]
cmdlib._ReleaseLock: Do nothing if no locks are owned
The locking library doesn't like it when “release()” is called on
a lockset or lock which isn't held by the current thread. Instead
of modifying the library, which could have other side-effects,
this rather simple change avoids errors when a LU simply tries to
release all locks, even when it doesn't own any at a certain level.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 15 Nov 2011 06:21:34 +0000 (07:21 +0100)]
Use resource lock when setting node parameters
Also acquire instance and resource locks in shared mode (see comment).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 3 Nov 2011 16:12:19 +0000 (17:12 +0100)]
Use node resource lock for replacing instance disks
If early-release is not used, the resource lock is kept while waiting
for disks to sync.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 3 Nov 2011 16:19:30 +0000 (17:19 +0100)]
Hold node resource locks while setting instance parameters
Important for when disks are converted. Release locks once they're not
needed anymore. Make liberal use of assertions.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 3 Nov 2011 16:08:26 +0000 (17:08 +0100)]
Hold node resource lock while moving instance
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 3 Nov 2011 16:04:05 +0000 (17:04 +0100)]
Acquire node resource lock when removing instance
Removing an instance affects available disk space and memory.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 3 Nov 2011 15:58:00 +0000 (16:58 +0100)]
Use node resource lock when recreating instance disks
Recreating disks conflicts with other disk operations, therefore the
node resource lock must be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 3 Nov 2011 15:40:06 +0000 (16:40 +0100)]
LUClusterRepairDiskSizes: Use node resource locks
Since this doesn't really touch the node, but it conflicts with e.g.
growing a disk, the resource lock must be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 2 Nov 2011 15:55:33 +0000 (16:55 +0100)]
LUInstanceGrowDisk: Use node resource lock
Also add one more feedback line. Downgrade instance lock to shared mode
while we're only waiting for disks to sync. The node lock is released
when not needed anymore.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 2 Nov 2011 15:33:46 +0000 (16:33 +0100)]
LUInstanceCreate: Hold node resource lock
The node resource lock is released once the disks are in sync (that is,
after wiping).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 14 Nov 2011 13:12:08 +0000 (14:12 +0100)]
LUNodeQueryvols: Acquire all locks in shared mode
Nothing is being written to.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 14 Nov 2011 13:11:48 +0000 (14:11 +0100)]
LUNodeQueryStorage: Acquire all locks in shared mode
Nothing is written to.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 14 Nov 2011 09:36:50 +0000 (10:36 +0100)]
cmdlib: Share lock in LUInstanceConsole
No writes are being done.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Fri, 11 Nov 2011 17:04:55 +0000 (18:04 +0100)]
Document OpNodeMigrate's result for RAPI
- Commit
b7a1c8161 changed the LU to generate jobs
- Mention documented results in NEWS
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 3 Nov 2011 16:22:34 +0000 (17:22 +0100)]
LUNodeQuery: Call implementation's DeclareLocks function
Just in case we ever add locks for querying nodes. Currently
_NodeQuery's DeclareLocks is a no-op function.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>