Use 'ArrayObject' in Ganeti.RPC instead of manual encoding
Before most of the calls were implemented manually, ignoring anygenerated code for maniupulating JSON. Using 'ArrayObject' we canautomate the process.
Signed-off-by: Petr Pudlak <pudlak@google.com>...
Move 'DictObject' and 'ArrayObject' to Ganeti.JSON
The type classes are more closely related to JSON and for using themit should not be necessary to import Ganeti.THH.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add a typeclass for type serializable as a JSON array
This is in particular needed for RPC requests which are serialized thisway and therefore using `showJSON` on the generated objects doesn'twork.
The fiels are serialized in the order they're declared, together with...
Document 'fieldExtraKeys' in 'Field'
Remove toDist... functions completely
Since thre already is `DictObject` class with `toDict`, there is no needfor them, just polluting the name space.
Enhance DictObject and refactor loading/saving TH functions
Since there is already `DictObject` type class which could share codewith generated instances of `showJSON`, the generated code is split intotwo parts: One that converts between `[(String, JSValue)]` and the data...
Write lock state to disk after updates
Make use of asynchronous worker to save to log status after everyupdate. The answer to the query is only returned, once the stateis saved on disk. Note that replication of the lock status is notnecessary, as all jobs will be gone if the master node goes away....
Provide an asynchronous task for writing lock status
It will be used to persist the state of the lock allocation ondisk, allowing locks to survive reboots of WConfD.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Add Ganeti Lock Hierarchy
Make Ganeti.Locking.Locks.GanetiLocks present the full Ganetilock hierarchy. This is in accordance with the current Pythonimplementation.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>...
Serialize locks as JSON strings
...as the current Python implementation does as well.
Export opportunistic lock union through WConfd
Make WConfD provide a function to opportunistically allocatelocks.
Support name generation for pairs
Add support for naming of python variables if the typeis a pair.
Note: on older versions of haskell, Q is not an instance ofApplicative, hence we use the do-syntax here.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Export lock intersection through WConfD
Export the functionality to restrict the owned locks to a given set.
Export freeLocks through WConfD
Export the functionality to free all locks of a givenowner.
Add lock modification without result to WConfdMonad
Enrich the structure of WConfdMonad by a conveniencewrapper around modifyLockAllocation for the case thatthe modification always succeeds and does not produceany result.
Export updateLocks through WConfD
Export the functionality to update the locks of a givenowner through WConfD.
Add serialization for LockRequest GanetiLocks
Add a module describing how abstract requests, like lock requests arepassed over the wire to WConfD.
Export listing of locks
Make WconfD answer requests about the locks of a specificowner.
Support reading the lock allocation in WConfdMonad
As opposed to the atomic modify-and-read provided my modifyLockAllocation,no storing is necessary here.
Merge branch 'stable-2.11' into master
Fix generic uncurrying
in the case of n>1, the n-ary uncurry function is of the form\f (x_1, ..., x_n) -> ... rather than \f x_1 ... x_n -> ...
Allow cluster mac prefix modification
Extend LUClusterSetParams to allow the modification of the clustermac-prefix setting in 'gnt-cluster modify' command.
This fixes part of issue 239.
Signed-off-by: Dimitris Bliablias <bl.dimitris@gmail.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Show mac prefix setting in gnt-cluster info
Include mac-prefix setting in the output of 'gnt-cluster info'command.
Locks are owned by job id file path pairs
...according to our updated design.
Remove the client state from WConfD monad
As we decided to keep our protocols stateless, the clientstate is no longer needed.
Have SubmitManyJobs add entries to the reason trail
Not only SubmitJobToDrainedQueue (and therefore SubmitJob) but alsoSubmitManyJobs has to add "gnt:opcode:*" entries to the reason trail.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add reason trail pickup constant
Add a constant for the reason trail, representing the pickup of a job from thedisk.
Also, refactor a bit the other constants so that part of the definition can beshared in a hierarichical fashion.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Have LuxiD add the "gnt:opcode" reason trail entry
The entry used to be added in jqueue.py, but after switching the queuemanagement from masterd to luxyd it had been lost. Now, make LuxiD responsiblefor adding it.
Add function for extending the reason trail in Luxid
The function will be used by the next commit.
Also, remove a few trailing whitespaces lying around the file.
Add missing constant for the name of WConfd
The constant is used when calling the daemon startup/shutdown scripts.
Add the --force-node option to WConfd
.. and fix the superfluous options list in ganeti-wconfd.hs
Add an option for disabling node checks at daemon startup
This is needed for WConfd, because during failover it needs to betemporarily started in order to update the configuration on the selectedmaster candidate.
While at it, fix the order of imports in OpCodes.hs
.. so that Ganeti imports are below library imports and orderedalphabetically.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Add a function for generating OpCode reason src. names
The function convert the opcode name to lowercase with underscores,strips the 'Op' prefix and prepends Constants.opcodeReasonSrcOpcode.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>...
Add a TH function for lower-cased stripped opcode names
The function strips the 'Op' prefix from a constructor name and convertsit to lower-case with underscores.
Generalize genConstrToStr to custom monadic functions
This will allow compile-time checks for constructor names.
Prefer unless over when (not...)
...and by doing so, please lint.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
At startup, make WConfD read the lock allocation from disk
On startup, make wconfd read a the state of the lock allocation from disk,instead of starting from an empty allocation.
Note that, as opposed to the configuration, no finding a savedlog configuration is not an error; e.g., the node might be master...
Provide path to persist locks
As we aim to handle jobs as separate processes with lockshandled by WConfD, we need to persist the locks to ensurethey survive a restart of that deamon. Saving only on masterdaemon is enough, however.
Add a function to load lock allocation from disk
Lock allocation are stored on disk as their JSON representation.
Make Locks a JSON instance
As requests to acquire or release locks come viainter-process communication, we need a serializationof the locks data type.
All locks are serialised as JSON objects, where thekey "type" describes the type of lock, and the remaining...
Make LockAllocation an instance of JSON
In this way, we can serialize a lock state to disk, so thatlocks can survive a restart of WConfD.
Add query support for locks to luxid
While requests only get forwarded, it still helps toget luxid feature-complete with respect to master.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>Cherry-picked from commit a6e406ce376453e90e598c7be68809d6a7bd7d41...
Provide fields for lock queries
For luxid to be feature-complete with respect to masterd, italso needs to answer requests about locks. This includes knowingthe fields available for locks.
Export reading and writing the configuration using RPC
Note that currently these functions lack any kind of locking andconcurrent calls to 'writeConfig' will result in lost changes in theconfiguration.
Add a module for functions specific to config. management
.. and include ConfigData in WConfd's ConfigState.
Currently it includes functions for asynchronous saving theconfiguration and reading and (over)writing it in the daemon state.
Load the configuration from the file at daemon startup and configure an...
Add more general conf. handling functions to WConfd's Monad
Add functions for reading and writing the configuration. Writingtriggers the asynchronous worker which saves the configuration into themaster file. The actual action that saves the configuration is passed as...
Add functions for saving the configuration to Ganeti.Config
.. so that it can be not only read, but also written to the masterconfiguration file.
Add a module for creating and using asynchronous workers
Each such worker has its own background thread and performs a giventask when triggered. Triggers arriving when the worker is processing areaccumulated and processed together in a subsequent worker action....
A separate module for functions for atomic file operations
Utils is getting too big, so better split this new set of functions intoa separate sub-module. This also allows us to use ResultG there.
Add a function for lifting ResultT from a MonadBase
Often we have `ResultT e IO a` as the return type of our primitivefunctions and we need to lift them into a monad stack that is also basedon IO. This function accomplishes this by lifting a ResultT in a base...
Add a type alias for "ResultT GanetiException IO"
This type occurs often in the upcoming code so it's worth having analias for it.
It's included in Errors instead of Utils, because it depends onGanetiException, and importing Errors from Utils causes a cyclic...
Add logging functions lifted into MonadBase
Calling logging functions in a monad transformer stack requiresinstances for all the transformers.This patch adds a module with functions named just as in Ganeti.Logging,but lifted into MonadBase. This allows using them in any transformer...
Add a MonadLog instance for strict StateT and IdentityT
This will be used for asynchronous workers, for which it's convenient touse StateT to keep their internal state.
Merge branch 'stable-2.10' into stable-2.11
Merge branch 'stable-2.9' into stable-2.10
Support opportunistic lock allocation
Add a function to opportunistically allocate as many locks of a givenset as possible. This is equivalent to sequentially try the locks inincreasing lock order after restricting to those locks where therequired owner state is higher than the currently held one....
Add a convenience function to restrict locks to a given set
As it is an operation the locking daemon is supposed to provide,add a function restricting the locks of a user to a given listand release all other locks.
Enforce the order restrictions on group locks
Disallow requests for an exclusive lock, if the owner alreadyholds a shared (and only shared) lock on a group lock that lockbelongs to.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Make hbal deal with no-LVM storage space properly
Since 2.6, hbal crashes when used on a cluster where noLVM storage is enabled at all. The problem is that italways queries for fields that only sometimes makesense for certain types of storage. This patch will...
Efficiently verify consistent lock requests
Most requests for lock updates are consistent and mentionevery lock only once. So verify this property efficientlyby comparing lengths. Only if the length do not coincidego through the actual (quadratic) comparison to find a...
Support lock implication in allocation
Make the module abstractly handling lock allocationhonor the additional restrictions caused by lockimplications.
Add a convenience function to free all locks of an owner
While freeing all locks of a single owner can easily be definedout of listLocks and updateLocks, it is worth having this functionin its own right. For example, it will be needed when the death of...
Make GanetiLocks an instance of Lock
Add a type class describing types that can serve as locks
Besides the lock order and Show, an additional datum is needed in orderto describe lock inclusion, used, e.g., for group locks. So add a typeclass describing this property.
Add missing documentation to exported function listLocks
The module Ganeti.Locking.Allocation exports the function listLocks,hence a documentation string is required. Add it.
Conflicts: NEWS: take both additions configure.ac: ignore suffix bump...
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Constant for instance communication network mode
Create a new constant to hold the instance communication network modeas this constant will be necessary during the QA, and update thegeneral documentation about the constants related to the instancecommunication mechanism....
Add '-c | --instance-communication' flag to instance modify
Fix param name to conform to convention of optional params
The opcode parameters can be optional and parameters that are optionalhave their names prefixed by 'pOpt'. This patch fixes with parameterto conform to this convention.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>...
Fix whitespace and typos in comments
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Disable deprecation warnings in PyRPC
Since on Debian Squeeze we have TH version 2.4, we need to use 'report'instead of 'reportWarning/reportError'. However, 'report' is deprecatedin later versions, therefore we need to disable the warning.
The warning doesn't manifest on Debian Squeeze or Wheezy, but causes...
Make the lock allocation part of WConfD's state
As WConfD is supposed to be the authoritative source forconfiguration and locks, it needs to have access to the lockstate as well.
Add a module for the Ganeti lock structure
Already provide a module holding the structure of the locksavailable in Ganeti, so that the overall structure of WConfDcan be implemented. The actual locks still have to be added.
Derive Ord JobId
Make the type JobId an instance of Ord. While the order itselfdoes not matter, it is a prerequisite to be an instance of Ordto be used as the type of lock owners.
Add function describing lock updates
Add the pure part of the mechanism of updating locks. To allowfor efficient waiting on locks, return the set of owners of the lockson which the operation is blocked.
Add a function to leave the list monad
The list monad provides convenient syntax for non-deterministicalgorithms. Add a function leaving that monad with this intuitionin mind.
Add data structure representing current lock status
To allow for jobs as processes, a central daemon (wconfd) willhandle allocation and release of locks. Add an appropriate datastructure to describe the current status of the locks.
Use node UUID as client certificate serial number
It turns out, that some implementations of OpenSSL are morepedantic in checking the certficates than others. In thisparticular case, the SSL connection could not beestablished when the serial number of the certificates...
Revert "Disabling client certificate usage"
This reverts commit 45f75526b848, which was introduced totemporarily disable the implementation of SSL clientcertificates. As this patch series fixes the reason forthe disabling, we are rolling back the patch....
Add functions for atomic operations on files
Function 'atomicUpdateFile' extends 'atomicWriteFile' that allows anyaction to be run on a temporary file.
Function 'atomicUpdateLockedFile' additionally locks the original fileusing flock and checks if its state conforms to the last one....
Add mtime/ctime to ConfigData
It's used on the Python side, it's been missing on the Haskell side.
Re-export liftIO from BasicType
This makes usage of ReaderT's liftIO more convenient.
Generate WConfD Python stubs from Haskell
The generation is included in hs2py.hs, together with otherHaskell to Python code generators.
Generate Python stubs by inspecting functions using TH
The names of the arguments of generated Python methods are derived fromHaskell types to be as descriptive as possible.
Add the WConfD daemon itself
The daemon exposes the declared functions in Ganeti.WConfd.Core to RPCclients (currently just 'echo').
Add the WConfD daemon to build configuration files
Also list it in the Haskell datatype, constants, Python constants andtest configuration.
Add a module for WConfd core functions
There are no exported functions yet, except for `echo`, which justreturns its argument back to a client.
Add a MonadLog instance for strict variant of RWS
This allows to log in monad stacks containing RWS.
Add a monad for running all WConfD functions
This monad encapsulates working with the daemon and client state, aswell as failures and IO operations.
Add a generic RPC module
The main function in this module takes a list of names of functions anduses TH to create an RPC server, represented as a value of type Handler(from Ganeti.UDSServer). This can be readily used to create a daemonthat dispatches RPC requests to the given functions....
Add common functions for working with TH's Type
In particular, functions for extracting types of arguments from afunction type, and for uncurrying functions with an arbitrary number ofarguments.
Another workaround for GHC linking error "unknown symbol"
This error appears after adding the dependency to lifted-base.See also #683 and ee7caf27.
I surmise that something like this is happening: Some code, that usesthe library, is only referenced through Template Haskell. A probable bug...
Add instances of MonadBase and MonadControl for ResultT
This allows to use lifted operations like 'fork' or 'bracket' insideResultT.
Allow clients of UDSServer to use different monads
.. as long as they're instances of "MonadBaseControl IO" and "MonadLog".This allows the UDSServer to call functions like "fork" within monadssuch as "ResultT e IO" or "ReaderT IO".
Merge branch 'stable-2.8' into stable-2.9
Fix integer overflow problem in hbal
waitForJobs in src/Ganeti/Jobs.hs has an integer overflow that (at least onamd64) causes it to break after waiting for ~10 minutes. This results in hbalsleeping forever (when compiled with squeeze's ghc 6.12.1) or crashing (when...