History | View | Annotate | Download (61.1 kB)
Replace frozenset with compat.UniqueFrozenset
This is not a trivial s/frozenset/compat.UniqueFrozenset/, but ratheronly replaces “frozenset” where appropriate. Most of the places are“static” information that doesn't change after the module has beenloaded....
Add opportunistic locking to GanetiLockManager
Just forwarding the parameter, nothing more.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
locking: Implement opportunistic locking in LockSet
This patch adds a new parameter to “LockSet.acquire” named“opportunistic”. When enabled the lockset will try to acquire as manylocks as possible, but it won't wait for them (with the exception of thelockset-internal lock in case the whole set is acquired). This is...
locking: Change locking order, move NAL after instances
Some opcodes, for example LUInstanceFailover and LUInstanceMigrate,can't know whether they need to acquire all nodes until they have a lockon the instance. In turn they would have to acquire the node allocation...
locking: Method to check if LockSet is fully acquired
A new method is added to check whether the LockSet-internal lock isheld. This is the case after LockSet.acquire was called withlocking.ALL_SET.
Unit tests are updated, including one where the list of names must be...
Add new lock level for node allocations
The new lock is similar to the BGL in the sense that it has its ownlevel and there is only one. It is called “node allocation lock”.Logical units will use it to synchronize with instance creations, whichin turn will start using opportunistic locks on nodes....
locking.LockSet: Replace boolean parameter with constants
Upcoming changes will add opportunistic locking to “locking.LockSet”.Doing so will require additional code in “LockSet.__acquire_inner”, atwhich point the existing “want_all” parameter does not always apply....
locking: Use frozenset instead of utils.UniqueSequence
In this case “frozenset” is good enough as the result's order doesn'tmatter--it is the input to “sorted” (“utils.UniqueSequence” preservesthe order). “frozenset” is ca. 25% faster for this use-case....
Basic IP pool management logic
Implement LUs for corresponding opcodes: * LUNetworkAdd: - Check for IP validity - Reserves all necessary IPs - Create new Network config object * LUNetworkRemove: - Checks if connected to any nodegroup - Remove a Network config object...
locking: Simplify condition
locking: Don't schedule pending acq. for short timeout
Scheduling a pending acquisition is relatively expensive and lot of codeis involved. Unless there is already one, a new pipe needs to be opened.Data structures need to be updated as well, only to be undone shortly...
Documentation for the NODE_RES level
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Bump pep8 version to 1.2
Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:
- bump version in the docs- silence some new checks that are wrong due to our indent=2 instead of 4- fix lots of errors in the code where the indentation was wrong by 1...
Merge branch 'devel-2.5'
locking: Remove unused OldStyleQueryLocks
No longer used after commit 090377807.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
locking: Notify only once on release
Don't notify for every released lock in shared mode. The last one isenough.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
locking: Handle spurious notifications on lock acquire
This was already a TODO since the implementation of lock priorities inSeptember 2010. Under certain conditions a waiting acquire can benotified at a time when it can't actually get the lock. In this case it...
locking: Fix lock deletion with timeout
While working on another SharedLock fix I realized timeouts on lockdeletion don't work very well if the timeout actually expires. Thispatch fixes the issue and adds a new unittest.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Replace single- with double-quotes
In at least two cases "%s" is replaced with str(), too.
locking: Add “__repr__” to SharedLock and PipeCondition
These help when debugging.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Andrea Spadaccini <spadaccio@google.com>
locking: Allow checking if lock is owned in certain mode
With this patch the “LockSet” and “GanetiLockManager” classes have a newfunction to check if a single or a group of locks (at a certain level)have been acquired in a specific mode. This will be used for additional...
locking: Make some aliased methods public
Some methods, such as “_is_owned” and “list_owned”, have been aliased tomake them public for a while now. This patch makes the actualimplementation public.
SharedLock's “is_owned” needs to be aliased to “_is_owned” to remain...
locking: Make level names consistent, add check
- Change all lock level names to their singular form- Add assertion checking consistency between level names and lockset name
Add new lock level for node resource modifications
This is in preparation for implementing a new resource model.
DeprecationWarning fixes for pylint
In version 0.21, pylint unified all the disable-* (and enable-*)directives to disable (resp. enable). This leads to a lot ofDeprecationWarning being emitted even if one uses the recommendedversion of pylint (0.21.1, as stated in devnotes.rst)....
Fix lint errors
It turns out that the only use of the operator module was foritemgetter, so patch eb62069e should have removed that import too.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Add two more compat functions
operator.itemgetter(0) → fstoperator.itemgetter(1) → snd
snd is not used yet, but it makes sense to add both.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Make lock monitor more versatile
With this change it'll be possible to register other lock informationproviders. One usecase for this are job dependencies, which can be shownin the output of “gnt-debug locks”, too.
The lock monitor is changed to accept more than one return value from...
locking.GLM: Allow adding locks to monitor
This will be used for exporting job dependencies throughthe lock monitor.
Make SharedLock._is_owned public
This will be useful for assertions. GanetiLockManager._is_owned isexported, too.
SharedLock: Implement downgrade from exclusive to shared mode
If a job needs to modify a resource and then wait for a result, it mustacquire the resource lock in exclusive mode. In some cases it would bepossible to only have a shared lock for waiting. Until now it was not...
Show locksets in lock monitor
When all locks contained in a set are acquired, the lockset's internallock is acquired with the same mode. With this patch the internal lockwill show up on the lock monitor, named e.g. “instances/[lockset]”.
locking: Make parameter to condition's wait() positional
It is always used in the locking code. Unittests are updated.
SharedLock: Avoid acquires from sneaking in while notifying
In some rare cases new shared acquires could sneak in through thecondition cached in “__pending_shared” while the code was stillnotifying acquires. This was only working because such a condition...
locking: Export “list_owned” from lock manager
This is analog to “is_owned” and will be used for assertions.
locking: Fix race condition in lock monitor
In some rare cases it can happen that a lock is re-created very soonafter deletion, while the old instance hasn't been destructed yet. Insuch a case the code would detect a duplicate name and raise anexception....
Convert “gnt-debug locks” to query2
Locks can now be queried using “Query(what="lock", …)” over LUXI.
locking: add nodegroup lock level
This also changes masterd to initialize the ganeti's manager with thecurrent list of nodegroup uuids, and updates unittests
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Merge branch 'devel-2.3'
Conflicts: NEWS: Trivial
locking: Clarify message for removed locks
Just being told that a lock doesn't exist can be confusing. One casewere this happens is when a job (e.g. instance modify) waits for a jobremoving the instance (e.g. export with remove).
Move locking.RunningTimeout to utils
As we need this functionality in other places than just locking it makessense to move it to utils rather than keeping it in locking
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
GanetiLockManager, remove default values
The nodes and instances parameters to the constructor are mandatoryanyway, as a value of None will fail when creating the LockSet. Ratherthan fixing this adding code lines, since we never used the defaultvalue, let's remove them and require that the parameters are passed....
locking: Implement priority in Ganeti lock manager
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
locking: Don't set default priority as keyword default
This allows users of these classes to simply pass None if they want to use thedefault value (the actual default is an internal constant), instead ofdynamically constructing the keyword arguments.
Merge branch 'devel-2.2'
locking.SharedLock: Update class docstring
This was already outdated when the initial version of SharedLock was addedin commit 162c1c1f1 (February 2008).
locking: Implement priorities in SharedLock and LockSet
For proper support of job priorities, jobs' locks need to respectpriorities. Otherwise it could happen that a job with a lower prioritycould get a lock before a job with a higher priority (depending on...
Show list of pending acquires in “gnt-debug locks”
This is accomplished by keeping a list of waiting threads insteadof just their number inside the lock-internal condition. A fewother tweaks to the output format are also made.
Add simple lock monitor
This patch adds an initial implementation of a lock monitor, accessiblefor the user through “gnt-debug locks”. It currently shows all resourcelocks: BGL, nodes and instances. Config and job queue locks could beshown too, but wouldn't be of much help. The current owner(s) and mode...
locking.LockSet: Use function to get member lock name
Implement lock names for debugging purposes
This patch adds lock names to SharedLocks and LockSets, that can be usedlater for displaying the actual locks being held/used in places where weonly have the lock, and not the entire context of the locking operation....
ssynchronized: act on a class member
The ssynchronized decorator takes the lock to act on in input.With this change we allow a string to be passed, and if so we assume thefunction it protects is a class method, and we act on the member of theclass itself named as the string we got....
Remove locking._CountingCondition
This class is unused and untested. We must have forgot it around.
_BaseCondition: allow saving/restoring state
SharedLock _acquire_restore and _release_save
If a shared lock is used inside a condition, we need to make sure thatit's reacquired in the same way as it was originally, after the wait.
Add separate module for backported language functionality
utils.py, where they were before, is already huge.
Merge remote branch 'origin/stable-2.1' into devel-2.1
Fix bug introduced in commit 413b747
While commit 413b747 fixed the issue of poll(2) returning toosoon, it didn't work when the poll(2) call should've beenblocking. This is now fixed and verified.
Fix locking bug causing high CPU usage
Iustin Pop noticed unusually high CPU usage with 2.1's masterdaemon, even with very simple opcodes like OP_TEST_DELAY. Asit turns out, we inadvertently passed seconds as millisecondsto a call to poll(2). Due to the way the loop around the call...
locking: add/fix @type information
This patch missing @type information for all public methods, modifiesone to conform to the rest, and removes some information from @paramwhen it's been expressed in @type.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Fix slots definitions
According to http://docs.python.org/reference/datamodel.html#slots
locking: Fix race condition in LockSet
This patch fixes a race condition when acquiring all locks ina LockSet instance. The list of lock names needs to be sortedto guarantee a consistent locking order, but the names were notsorted when acquiring all locks in the set....
locking: Append to list outside error handling block
locking: Don't fail in error handling if lock isn't owned
In case an exception was thrown while acquiring the lock, not necessarily allowned locks are also really acquired. Before this change, an exception could bemasked by another exception thrown here. There is no good clean-up strategy...
Add targeted pylint disables
This patch should have only:
- pylint disables- docstring changes- whitespace changes
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Olivier Tharan <olive@google.com>
Merge branch 'devel-2.0' into devel-2.1
Add targetted pylint disables
This patch adds targeted pylint disables, where it makes sense (eitherdue to limitations in pylint or due to historical usage), and also a fewblanket ones in rapi where all the names are… “different”.
Signed-off-by: Iustin Pop <iustin@google.com>...
Clarifiy some more wide pylint disables
This removes/updates some module-wide pylint disables.
Remove quotes from CommaJoin and convert to it
This patch removes the quotes from CommaJoin and converts most of thecallers (that I could find) to it. Since CommaJoin does str(i) for i inparam, we can remove these, thus simplifying slightly a few calls....
locking: Convert pipe condition to new timeout class
locking.LockSet: Move timeout calculation to separate class
This class can also be used by mcpu.
locking, mcpu: Ensure timeout is always >= 0.0
locking.LockSet: Improve assertions
locking: Factorize LockSet.acquire
By moving the main code of LockSet.acquire to its own functionwe reduce the code complexity a bit and clarify the exceptionhandling.
This also fixes a case where a lock acquire timeout wasn'thandled correctly, leading to obscure error messages....
Code and docstring style fixes
Found using pylint and epydoc.
locking.LockSet: Implement acquire timeouts
The timeout passed to LockSet.acquire() is measured over all lock acquires. IfLockSet.acquire fails to acquire all requested locks within the specifiedamount of time, all locks are released again and the acquire fails....
More locking tests race conditions fixes
There were more race conditions. By adding a notify function toSharedLock.acquire we can prevent them.
SingleActionPipeCondition =~ s/Action/Notify/
With this patch we simplify usage on the SingleActionCondition (whichwasn't a condition at all) by making it a real condition. This way wecan just wait() on it, or notifyAll() as we would on a normal one. The...
Abstract "base" condition code in a separate class
Each condition has an underlying lock, the acquire and release methods,and a few helper methods to check that it's called in the proper way.
Abstract them to a separate class so we can have more than one without...
locking.SharedLock: Fix bug in delete function
SharedLock.__acquire_unlocked uses keyword parameters. Just passingthe timeout would set the “shared” parameter.
Rename LockSet.acquire parameter “blocking” to “timeout”
Also remove the “blocking” parameter from LockSet.remove andGanetiLockManager.remove. There's no point in implementing timeouts on removalunless we need them.
Change SharedLock to new pipe(2)-based condition
Add _PipeCondition class
_PipeCondition is a condition implemented using pipe(2) and poll(2).It allows the implementation of timeouts without using a busy-wait loopwith time.sleep.
Unlike Python's built-in threading.Condition class and to save filedescriptors and an internal queue, it can only be used to notify...
Add _SingleActionPipeCondition class
This class will be used as a basic block for pipe(2)-basedconditions. Upon initialization it creates a pipe and can benotified once (hence the “single action” in the name). Acallable helper class is used to wait for notifications....
SharedLock: implement timeouts
This patch greatly simplifies the SharedLock code and implementstimeouts for the acquire() and delete() functions. A wrapper aroundPython's threading.Condition class must be used to ensure threadsafety when check whether there are any waiters left....
locking: Acquire SharedLock in shared mode in separate function
This is for some more symetry withSharedLock.__exclusive_acquire.
locking: Add level name dict
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Olivier Tharan <olive@google.com>
locking: Don't swallow exceptions
This is an indentation bug.
Fix some typos
locking.LockSet: don't modify input arguments
Currently LockSet.acquire() sorts in place it's input argument if it's alist. This is not good, since callers might depend on a specificordering of the input data, and this is a 'hidden' modification.
We fix it by simply using a sorted copy, instead of sorting in place....
Re-wrap some lines to keep them under 80 chars
This non-code change rewraps some lines in locking.py to keep them under80 chars.
Reviewed-by: ultrotter
Fix epydoc format warnings
This patch should fix all outstanding epydoc parsing errors; as such, weswitch epydoc into verbose mode so that any new errors will be visible.
Reviewed-by: imsnah
LockSet: forbid add() on a partially owned set
This patch bans add() on a half-acquired set. This behavior waspreviously possible, but created a deadlock if someone tried to acquirethe set-lock in the meantime, and thus is now forbidden. ThetestAddRemove unit test is fixed for this new behavior, and includes a...
Fix typo in a locking.py comment
Add GanetiLockManager.is_owned function
This is a public version of the private function we already had.We don't just change the previous version because it had lots of usersin the library itself and in the testing code.
Fix LockSet._names() to work with the set-lock
If the set-lock is acquired, currently, the _names function will fail ona double acquire of a non-recursive lock. This patch fixes the behavior,and some lines of code added to the testAcquireSetLock test check that...
Add locking.ALL_SET constant and use it
Rather than specifying None in needed_locks every time, with a nicecomment saying to read what we mean rather than what we write, and thatNone actually means All, in our magic world, we'll hide this secretunder the ALL_SET constant in the locking module, which has value, you...
LockSet: allow lists with duplicate values
If a list with a duplicate value is passed to a lockset what the codenow does is to try to acquire the lock twice, generating adouble-acquire exception in the SharedLock code. This is definitely anissue. In order to solve it we can either forbit double values in a list...
Invert nodes/instances locking order
An implementation mistake from the original design caused nodes to belocked before instances, rather than after. This patch inverts the levelnumbering, changing also the relevant unittests and the recursivelocking function starting point....