Add an asynchronous task for distributing SSConf to nodes
The task computes the SSconf data and compares it to the last state thatwas successfully distributed. If it differs, issues the RPC call todistribute it. If any of the RPC calls fails, the state remains...
Add Haskell RPC call for distributing SSConf to nodes
The data is sent as a dictionary mapping file names to lists of lines.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Let the SSConf RPC server side handle lists
Since on Haskell side we represent SSConf as list of lines, let thePython side understand it as well.
Add a test for (de)serialization of Ssconf
.. to verify that readJSON . showJSON = Ok.
Add a module for constructing Ssconf from the configuration
This migrates the functionality from PythonConfigWriter._UnlockedGetSsconfValues to Haskell.
Add a data type for representing the complete SSConf
.. as a map from SSKeys to lists of strings.
Improve Haskell functions for managing the UidPool
Add a formatting function that is needed by Ssconf.Also expose 'ipFamilyToRaw', which is needed as well.
Add missing keys to Ganeti.Ssconf
These keys were probably introduced only in Python later and weren'tadded to the Haskell code.
Let WConfd distribute the configuration to MCs
.. and remove the distribution from lib/config.py
Function that distributes the configuration to MCs
Also refactor helper functions in WConfd/ConfigWriter into smaller onesand add helpers for RPC calls.
Add a function for listing online nodes to Haskell
.. and also simplify 'getMasterCandidates'.
Add the RPC call for uploading a file to Haskell
.. and also add a function that prepares the call from a given file.
Functions for conversions between POSIX time and ClockTime
This will be needed by the RPC calls to convert Haskell nativerepresentation to the one used in RPC.
Add reverse mappings to RuntimeEnts from UIDs to names
.. and GIDs to names. Also create a proper data type for it.
The reverse mapping to names is needed for properly translating localIDs to names for RPC calls.
Use 'ResultT' for 'getEnts' and handle all exceptions, not just when a...
Add a new RPC server call for uploading a single file
The server side processes the request exactly the same as for"upload_file".
Unlike "upload_file", the new call "upload_file_single" declares allrequired fields without requiring additional preprocessing....
Add more meaningful error messages to asserts in vcluster
.. to simplify debugging of RPC calls.
Create a data type for representing RPC compressed data
It is serialized as a JSON pair. The first element is a numberdescribing the compression type, the second element the actual data(in this case Base64+Zlib encoded).
On the Python side it is decompressed by backend._Decompress....
Add a module for handling compression
Since the Haskell 'zlib' library by default uses exceptions forsignalling errors, we need to use its internals to implement a proper,pure decompressing function.
Enhance logRpcResults to also return the list of errors
It is common to log all RPC errors and then check if there were any.Since 'logRpcResults' needs to filter the errors anyway, let it returnit.
Factor out the part that filters out errors into a separate function....
Use lenses for modifying DaemonState in WConfd
This removes the verbose code for record updating and leads to morepoint-free code.
Write the configuration only if it changed
.. to avoid expensive operation when they're not actually needed.Comparing the two objects in memory is much less costly thatdistributing the configuration.
Add a module for generated lenses for Ganeti.Objects
The module also adds type classes for lenses for common fields (serial,mTime, uuid and tags).
Splitting the lenses into a separate module avoids problems with unusedfunctions.
Signed-off-by: Petr Pudlak <pudlak@google.com>...
Add a module for custom generic Lens functions
The module currently re-exports Control.Lens and adds a function thatcreates lenses from data types by appending 'L' to its field names.
All Ganeti modules should import this module instead of Control.Lens...
Add 'lens' Haskell dependency
.. that will be used in WConfd
Add a helper for creating 'FileMode' fields
.. which will be used for RPC objects.
Move helpers for specific fields into a separate module
These helpers aren't needed by other code in Ganeti.THH and since themodule is already large, it'll be better to separate them into theirown.
No functionality is changed by this patch, just functions moved....
Add a helper to ConfigWriter for creating async. workers
In particular, stateful workers that can fail with an error, in whichcase the error is logged and the state remains unchanged.
Add Foldable/Traversable instances for GenericContainer
This makes working with it easier as it allows use of many standardfunctions.
Update style guide with respect to Haskell imports
With the number of imports increasing, our style hasevolved over time.
- We don't distinguish between standard and non-standard non-Ganeti libraries any more.
- To help the reader identify which library a function is...
Clean up RunWithLocks
This patch cleans RunWithLocks up a little bit by reducing the numberof delay function terminations, and using the QAThread class to ensureexceptions are thrown at the right time and in the right place.
Signed-off-by: Hrvoje Ribicic <riba@google.com>...
Add an exception-trapping thread class
To have better control over threads, this patch adds a helper Threadsubclass which captures any exceptions occurring for later use.
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Wait for delay to provide interruption information
The RunWithLocks test assumed that gnt-debug delay would have the infoneeded for interruption ready immediately after being run, and in somesituations this is not the case. This patch makes the test more patient...
Also clean up stale livelock files
Upon detection of a stale livelock, make WConfD alsotry to clean it up. We can safely ignore any failures,as the Ganeti resources are already freed, and the worstthing that can happen is an empty file left over in the...
Add Haskell dependency on 'lens'
.. which will be needed soon for WConfd.
While at it, also ensure we have the required version ofbase64-bytestring.
Refactor warnings issued by the latest 'hlint'
.. as the newest version of hlint adds additional checks
devel: Instal vanilla GHC7 in the squeeze schroot
Since the default GHC6 isn't usable for modern libraries and startingfrom wheezy there is GHC7 anyway, download and install the currentvanilla version as well as the latest Cabal.
Use the current up-to-date libraries....
devel: Make the built schroots rsync-able, if available
This speeds down transferring them to buildbots, at a negligible sizecost.
Use correct lockfile for gnt-debug wconfd
As jobs are currently running in masterd, use the masterd livelockfile.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Add utility to guess livelock file for an owner
As livelock files are constructed in a systematic manner,we can guess what the livelock file for a given owner is.While this will not necessarily work perfectly, it will beuseful to simplify direct debugging of WConfD....
Make masterd create a livelock file
...so that it can request resources from WConfd.
Rename setup_queue to setup_context in masterd
...as this function sets up a much richer context than justthe job queue, including the current lock management.
Add utilities for liveliness lock files
To request resources from WConfD, requesters have to providethe name of a file they own an exclusive lock on. In this way,their death can be detected. Add utility functions to obtainsuch a file name.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Ensure the existence of LIVELOCK_DIR
Add a path to store the lock files presented to WConfD
When requesting resources from WConfD, a file has to bepresented where an exclusive lock is owned on, so thatWConfD can detect when the requester dies. Add a pathto a directory where these files are kept in....
In WConfD run a cleanup task
Add the cleanup task for locks of died owners to WConfD,as a separate thread.
Provide a periodic lock cleanup task
Provide a task that periodically checks for all locksowners whether they are still alive. If not, it cleansup their locks.
Add a constant for the polling of jobs death
Add a constant for the time interval at which WConfD checks forresource owners that have died and cleans up their resources.
Verify that all lock owners indeed own a lock
Add a test verifying that the list of lock owners is nottoo big in the sense that every subject contained in thatlist owns at least one lock.
Verify the list of lock owners is complete
In other words, verify that if all the lock owners release theirlocks, the empty lock allocation is reached.
Add a function to determine the lock owners
To allow for clean up, add a function that provides the listof all the owners holding at least one lock.
Add a function to detect death of a job
With jobs running in processes different from the lock management soon,we need to detect if a job holding locks dies, in order to release theresources.
Merge branch 'stable-2.11' into master
Convert int to float when checking config. consistency
When reading the configuration file from RPC JSON, values without afloating point are parsed as 'int', not as 'float', and later theconsistency check fails.
This patch adds an automatic conversion from 'int' to 'float' during...
Update the NEWS file
Add an expected block option to RunWithLocks
To compensate for the cases where a QA test is supposed to block whena lock is present, add an additional option showing whether blocking issupposed to happen or not.
Track if a QA test was blocked by locks
This patch adds threading to the RunWithTests function, allowing onethread to execute the QA test, and the other to monitor if it is beingblocked by locks set up during the test. If it is, terminate theblocking job, and let the QA continue, reporting the test failure at...
Add option to turn off command logging to GetCommandOutput
The command logging performed by GetCommandOutput can be distractingand unwelcome. This patch adds a switch allowing the command output tobe muted as needed.
Add an option for disabling the SSH multiplexer in QA
The SSH multiplexer is used by default in every connection made, andwhen multiple threads are used, this causes strange issues when runningthe QA locally. This patch allows the multiplexer to be bypassed,...
Add a RunWithLocks QA utility function
This patch adds a QA utility function that acquires a set of locks, andattempts to run a given function with the locks in place. Should thegiven function block, this function does not detect this - laterpatches will address the issue....
Sort imports in qa_rapi.py
This patch sorts the imports of qa_rapi.py to make things a little bitmore orderly.
Align timestamps in gnt-job info
This patch aligns the timestamps output as a part of gnt-job info, andperforms minor refactorings in the process.
Add alignment support to PrintGenericInfo
Aligning dictionary entries makes no difference to a YAML parser, butmakes the output much easier to read and compare. This patch adds thepossibility of specifying alignment groups to ordered dictionaryentries....
Make gnt-job info output valid YAML
This patch changes gnt-job info to use standard functions defined incli.py, and output valid YAML.
Make PrintGenericInfo handle tuples better
The PrintGenericInfo function in cli.py did not handle tuples ascontainers of items, making it impossible for these to be deserializedautomatically when a YAML parser is used. This patch adds separatehandling of tuples, including inlining them for readability when...
Refactor move-instance QA test
The move-instance QA test will have to be changed in the followingpatches to allow testing opportunistic locking.
This patch retains the same functionality as before, but allows aniallocator to be used, and splits the move back and forth into two...
Make gnt-debug delay interruptible
The gnt-debug delay command could be useful as a means of acquiringlocks for testing purposes. In practice, to be useful it should beinterruptible, otherwise we risk race conditions or long delays.
This patch follows the examples of the move-instance command and the...
Add new log type
Following the example of the job-queue test, the delay also defines alog message type of its own.
Add the interruptible option to gnt-debug delay
This patch allows the opcode option to be used through the gnt-debugclient.
Add the interruptible option to the delay opcode
To prepare for the use of sockets to interrupt a running delay opcode,this patch adds an option to the opcode for doing so.
Factor Unix domain socket creation into helper class
As the delay class will also have to start using domain sockets,extract the functionality into a helper class.
Fix minor accidental concatenation
Fix failing tests
The previously introduced changes caused some tests to fail. This patchalters the tests to expect different values, while still ensuringcorrect behavior.
Handle incorrect duration more elegantly
The previous version of the LUTestDelay opcode relied on the utilityfunction complaining about the negative duration. As this function hasbeen removed for now, do the check ourselves, and issue a moreappropriate exception....
Make gnt-debug delay command run in parallel
The gnt-debug delay command executes the delay first on the master, andonly then on all the other nodes, causing a significant delay. Thispatch makes the command treat the master as it would all other nodes....
Extend RAPI client constant unit test with error codes
The RAPI client is meant to be independent from the wider Ganeticodebase, but it still needs to use Ganeti constants. To enforce this,a unit test is used. This patch extends the unit test with error code...
Fix typo in RAPI client utility
Add variation to delays between creation attempts
Attempting to acquire node locks with a certain fixed delay could leadto unwanted synchronization with other processes competing for the samelocks. This patch adds some random variation to the attempts....
Implement opportunistic locking in the move-instance tool
This patch allows opportunistic instance creations to be attempted,with a delay between them giving nodes the opportunity to becomeavailable. After the given number of opportunistic tries has been...
Add options for opportunistic creation during moves
To start off the introduction of oppportunistic locking during instancecreation, this patch adds the options allowing the locking to beinvoked.
Remove duplicated '_CheckOSVariant'
It seems '_CheckOSVariant' was moved from 'ganeti.cmdlib.instance' to'ganeti.cmdlib.instance_utils' but the source was never deleted. Thispatch deletes the source copy if this function.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>...
Updateing release date to tomorrow
Since the release could not be made today, let's prepareit for tomorrow.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Revision bump for the 2.11.0~beta1 release
Set release date of 2.11 beta1
Set the release date of 2.11 beta1 in the NEWS file.
Use 'ArrayObject' in Ganeti.RPC instead of manual encoding
Before most of the calls were implemented manually, ignoring anygenerated code for maniupulating JSON. Using 'ArrayObject' we canautomate the process.
Test the instance 'ArrayObject Disk'
.. which is one of the few ones that use extra fields.
Add a helper function for testing 'ArrayObject' instances
It checks if 'fromJSArray . toJSArray' is equivalent to 'Ok'.
Move 'DictObject' and 'ArrayObject' to Ganeti.JSON
The type classes are more closely related to JSON and for using themit should not be necessary to import Ganeti.THH.
Add a typeclass for type serializable as a JSON array
This is in particular needed for RPC requests which are serialized thisway and therefore using `showJSON` on the generated objects doesn'twork.
The fiels are serialized in the order they're declared, together with...
Document 'fieldExtraKeys' in 'Field'
Remove toDist... functions completely
Since thre already is `DictObject` class with `toDict`, there is no needfor them, just polluting the name space.
Enhance DictObject and refactor loading/saving TH functions
Since there is already `DictObject` type class which could share codewith generated instances of `showJSON`, the generated code is split intotwo parts: One that converts between `[(String, JSValue)]` and the data...
Write lock state to disk after updates
Make use of asynchronous worker to save to log status after everyupdate. The answer to the query is only returned, once the stateis saved on disk. Note that replication of the lock status is notnecessary, as all jobs will be gone if the master node goes away....
Provide an asynchronous task for writing lock status
It will be used to persist the state of the lock allocation ondisk, allowing locks to survive reboots of WConfD.
Verify the intervall property of GanetiLocks
According to the specification in our design, if A is a grouplock and B belogs to this group, then all locks between A and Bin the lock order also must belong to A. Verify this.
Verify implication restriction on locks
As specified in our design, if A is a group lock and Bbelongs to A, then A must be earlier in the lock orderthan B. Verify this property.
Update Arbitrary GanetiLocks
With the extension of Ganeti.Locking.Locks.GanetiLocks to thefull lock hierarchy, also extend the instantiation of Arbitrary.
Add Ganeti Lock Hierarchy
Make Ganeti.Locking.Locks.GanetiLocks present the full Ganetilock hierarchy. This is in accordance with the current Pythonimplementation.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>...
Serialize locks as JSON strings
...as the current Python implementation does as well.
Export opportunistic lock union through WConfd
Make WConfD provide a function to opportunistically allocatelocks.
Support name generation for pairs
Add support for naming of python variables if the typeis a pair.
Note: on older versions of haskell, Q is not an instance ofApplicative, hence we use the do-syntax here.