Add a function for listing online nodes to Haskell
.. and also simplify 'getMasterCandidates'.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add the RPC call for uploading a file to Haskell
.. and also add a function that prepares the call from a given file.
Functions for conversions between POSIX time and ClockTime
This will be needed by the RPC calls to convert Haskell nativerepresentation to the one used in RPC.
Add reverse mappings to RuntimeEnts from UIDs to names
.. and GIDs to names. Also create a proper data type for it.
The reverse mapping to names is needed for properly translating localIDs to names for RPC calls.
Use 'ResultT' for 'getEnts' and handle all exceptions, not just when a...
Create a data type for representing RPC compressed data
It is serialized as a JSON pair. The first element is a numberdescribing the compression type, the second element the actual data(in this case Base64+Zlib encoded).
On the Python side it is decompressed by backend._Decompress....
Add a module for handling compression
Since the Haskell 'zlib' library by default uses exceptions forsignalling errors, we need to use its internals to implement a proper,pure decompressing function.
Enhance logRpcResults to also return the list of errors
It is common to log all RPC errors and then check if there were any.Since 'logRpcResults' needs to filter the errors anyway, let it returnit.
Factor out the part that filters out errors into a separate function....
Use lenses for modifying DaemonState in WConfd
This removes the verbose code for record updating and leads to morepoint-free code.
Write the configuration only if it changed
.. to avoid expensive operation when they're not actually needed.Comparing the two objects in memory is much less costly thatdistributing the configuration.
Add a module for generated lenses for Ganeti.Objects
The module also adds type classes for lenses for common fields (serial,mTime, uuid and tags).
Splitting the lenses into a separate module avoids problems with unusedfunctions.
Signed-off-by: Petr Pudlak <pudlak@google.com>...
Add a module for custom generic Lens functions
The module currently re-exports Control.Lens and adds a function thatcreates lenses from data types by appending 'L' to its field names.
All Ganeti modules should import this module instead of Control.Lens...
Add a helper for creating 'FileMode' fields
.. which will be used for RPC objects.
Move helpers for specific fields into a separate module
These helpers aren't needed by other code in Ganeti.THH and since themodule is already large, it'll be better to separate them into theirown.
No functionality is changed by this patch, just functions moved....
Add a helper to ConfigWriter for creating async. workers
In particular, stateful workers that can fail with an error, in whichcase the error is logged and the state remains unchanged.
Add Foldable/Traversable instances for GenericContainer
This makes working with it easier as it allows use of many standardfunctions.
Also clean up stale livelock files
Upon detection of a stale livelock, make WConfD alsotry to clean it up. We can safely ignore any failures,as the Ganeti resources are already freed, and the worstthing that can happen is an empty file left over in the...
Refactor warnings issued by the latest 'hlint'
.. as the newest version of hlint adds additional checks
In WConfD run a cleanup task
Add the cleanup task for locks of died owners to WConfD,as a separate thread.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Provide a periodic lock cleanup task
Provide a task that periodically checks for all locksowners whether they are still alive. If not, it cleansup their locks.
Add a constant for the polling of jobs death
Add a constant for the time interval at which WConfD checks forresource owners that have died and cleans up their resources.
Add a function to determine the lock owners
To allow for clean up, add a function that provides the listof all the owners holding at least one lock.
Add a function to detect death of a job
With jobs running in processes different from the lock management soon,we need to detect if a job holding locks dies, in order to release theresources.
Add new log type
Following the example of the job-queue test, the delay also defines alog message type of its own.
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Add the interruptible option to the delay opcode
To prepare for the use of sockets to interrupt a running delay opcode,this patch adds an option to the opcode for doing so.
Add options for opportunistic creation during moves
To start off the introduction of oppportunistic locking during instancecreation, this patch adds the options allowing the locking to beinvoked.
Use 'ArrayObject' in Ganeti.RPC instead of manual encoding
Before most of the calls were implemented manually, ignoring anygenerated code for maniupulating JSON. Using 'ArrayObject' we canautomate the process.
Move 'DictObject' and 'ArrayObject' to Ganeti.JSON
The type classes are more closely related to JSON and for using themit should not be necessary to import Ganeti.THH.
Add a typeclass for type serializable as a JSON array
This is in particular needed for RPC requests which are serialized thisway and therefore using `showJSON` on the generated objects doesn'twork.
The fiels are serialized in the order they're declared, together with...
Document 'fieldExtraKeys' in 'Field'
Remove toDist... functions completely
Since thre already is `DictObject` class with `toDict`, there is no needfor them, just polluting the name space.
Enhance DictObject and refactor loading/saving TH functions
Since there is already `DictObject` type class which could share codewith generated instances of `showJSON`, the generated code is split intotwo parts: One that converts between `[(String, JSValue)]` and the data...
Write lock state to disk after updates
Make use of asynchronous worker to save to log status after everyupdate. The answer to the query is only returned, once the stateis saved on disk. Note that replication of the lock status is notnecessary, as all jobs will be gone if the master node goes away....
Provide an asynchronous task for writing lock status
It will be used to persist the state of the lock allocation ondisk, allowing locks to survive reboots of WConfD.
Add Ganeti Lock Hierarchy
Make Ganeti.Locking.Locks.GanetiLocks present the full Ganetilock hierarchy. This is in accordance with the current Pythonimplementation.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>...
Serialize locks as JSON strings
...as the current Python implementation does as well.
Export opportunistic lock union through WConfd
Make WConfD provide a function to opportunistically allocatelocks.
Support name generation for pairs
Add support for naming of python variables if the typeis a pair.
Note: on older versions of haskell, Q is not an instance ofApplicative, hence we use the do-syntax here.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Export lock intersection through WConfD
Export the functionality to restrict the owned locks to a given set.
Export freeLocks through WConfD
Export the functionality to free all locks of a givenowner.
Add lock modification without result to WConfdMonad
Enrich the structure of WConfdMonad by a conveniencewrapper around modifyLockAllocation for the case thatthe modification always succeeds and does not produceany result.
Export updateLocks through WConfD
Export the functionality to update the locks of a givenowner through WConfD.
Add serialization for LockRequest GanetiLocks
Add a module describing how abstract requests, like lock requests arepassed over the wire to WConfD.
Export listing of locks
Make WconfD answer requests about the locks of a specificowner.
Support reading the lock allocation in WConfdMonad
As opposed to the atomic modify-and-read provided my modifyLockAllocation,no storing is necessary here.
Merge branch 'stable-2.11' into master
Fix generic uncurrying
in the case of n>1, the n-ary uncurry function is of the form\f (x_1, ..., x_n) -> ... rather than \f x_1 ... x_n -> ...
Allow cluster mac prefix modification
Extend LUClusterSetParams to allow the modification of the clustermac-prefix setting in 'gnt-cluster modify' command.
This fixes part of issue 239.
Signed-off-by: Dimitris Bliablias <bl.dimitris@gmail.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Show mac prefix setting in gnt-cluster info
Include mac-prefix setting in the output of 'gnt-cluster info'command.
Locks are owned by job id file path pairs
...according to our updated design.
Remove the client state from WConfD monad
As we decided to keep our protocols stateless, the clientstate is no longer needed.
Have SubmitManyJobs add entries to the reason trail
Not only SubmitJobToDrainedQueue (and therefore SubmitJob) but alsoSubmitManyJobs has to add "gnt:opcode:*" entries to the reason trail.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add reason trail pickup constant
Add a constant for the reason trail, representing the pickup of a job from thedisk.
Also, refactor a bit the other constants so that part of the definition can beshared in a hierarichical fashion.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Have LuxiD add the "gnt:opcode" reason trail entry
The entry used to be added in jqueue.py, but after switching the queuemanagement from masterd to luxyd it had been lost. Now, make LuxiD responsiblefor adding it.
Add function for extending the reason trail in Luxid
The function will be used by the next commit.
Also, remove a few trailing whitespaces lying around the file.
Add missing constant for the name of WConfd
The constant is used when calling the daemon startup/shutdown scripts.
Add the --force-node option to WConfd
.. and fix the superfluous options list in ganeti-wconfd.hs
Add an option for disabling node checks at daemon startup
This is needed for WConfd, because during failover it needs to betemporarily started in order to update the configuration on the selectedmaster candidate.
While at it, fix the order of imports in OpCodes.hs
.. so that Ganeti imports are below library imports and orderedalphabetically.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Add a function for generating OpCode reason src. names
The function convert the opcode name to lowercase with underscores,strips the 'Op' prefix and prepends Constants.opcodeReasonSrcOpcode.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>...
Add a TH function for lower-cased stripped opcode names
The function strips the 'Op' prefix from a constructor name and convertsit to lower-case with underscores.
Generalize genConstrToStr to custom monadic functions
This will allow compile-time checks for constructor names.
Prefer unless over when (not...)
...and by doing so, please lint.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
At startup, make WConfD read the lock allocation from disk
On startup, make wconfd read a the state of the lock allocation from disk,instead of starting from an empty allocation.
Note that, as opposed to the configuration, no finding a savedlog configuration is not an error; e.g., the node might be master...
Provide path to persist locks
As we aim to handle jobs as separate processes with lockshandled by WConfD, we need to persist the locks to ensurethey survive a restart of that deamon. Saving only on masterdaemon is enough, however.
Add a function to load lock allocation from disk
Lock allocation are stored on disk as their JSON representation.
Make Locks a JSON instance
As requests to acquire or release locks come viainter-process communication, we need a serializationof the locks data type.
All locks are serialised as JSON objects, where thekey "type" describes the type of lock, and the remaining...
Make LockAllocation an instance of JSON
In this way, we can serialize a lock state to disk, so thatlocks can survive a restart of WConfD.
Add query support for locks to luxid
While requests only get forwarded, it still helps toget luxid feature-complete with respect to master.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>Cherry-picked from commit a6e406ce376453e90e598c7be68809d6a7bd7d41...
Provide fields for lock queries
For luxid to be feature-complete with respect to masterd, italso needs to answer requests about locks. This includes knowingthe fields available for locks.
Export reading and writing the configuration using RPC
Note that currently these functions lack any kind of locking andconcurrent calls to 'writeConfig' will result in lost changes in theconfiguration.
Add a module for functions specific to config. management
.. and include ConfigData in WConfd's ConfigState.
Currently it includes functions for asynchronous saving theconfiguration and reading and (over)writing it in the daemon state.
Load the configuration from the file at daemon startup and configure an...
Add more general conf. handling functions to WConfd's Monad
Add functions for reading and writing the configuration. Writingtriggers the asynchronous worker which saves the configuration into themaster file. The actual action that saves the configuration is passed as...
Add functions for saving the configuration to Ganeti.Config
.. so that it can be not only read, but also written to the masterconfiguration file.
Add a module for creating and using asynchronous workers
Each such worker has its own background thread and performs a giventask when triggered. Triggers arriving when the worker is processing areaccumulated and processed together in a subsequent worker action....
A separate module for functions for atomic file operations
Utils is getting too big, so better split this new set of functions intoa separate sub-module. This also allows us to use ResultG there.
Add a function for lifting ResultT from a MonadBase
Often we have `ResultT e IO a` as the return type of our primitivefunctions and we need to lift them into a monad stack that is also basedon IO. This function accomplishes this by lifting a ResultT in a base...
Add a type alias for "ResultT GanetiException IO"
This type occurs often in the upcoming code so it's worth having analias for it.
It's included in Errors instead of Utils, because it depends onGanetiException, and importing Errors from Utils causes a cyclic...
Add logging functions lifted into MonadBase
Calling logging functions in a monad transformer stack requiresinstances for all the transformers.This patch adds a module with functions named just as in Ganeti.Logging,but lifted into MonadBase. This allows using them in any transformer...
Add a MonadLog instance for strict StateT and IdentityT
This will be used for asynchronous workers, for which it's convenient touse StateT to keep their internal state.
Merge branch 'stable-2.10' into stable-2.11
Merge branch 'stable-2.9' into stable-2.10
Support opportunistic lock allocation
Add a function to opportunistically allocate as many locks of a givenset as possible. This is equivalent to sequentially try the locks inincreasing lock order after restricting to those locks where therequired owner state is higher than the currently held one....
Add a convenience function to restrict locks to a given set
As it is an operation the locking daemon is supposed to provide,add a function restricting the locks of a user to a given listand release all other locks.
Enforce the order restrictions on group locks
Disallow requests for an exclusive lock, if the owner alreadyholds a shared (and only shared) lock on a group lock that lockbelongs to.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Make hbal deal with no-LVM storage space properly
Since 2.6, hbal crashes when used on a cluster where noLVM storage is enabled at all. The problem is that italways queries for fields that only sometimes makesense for certain types of storage. This patch will...
Efficiently verify consistent lock requests
Most requests for lock updates are consistent and mentionevery lock only once. So verify this property efficientlyby comparing lengths. Only if the length do not coincidego through the actual (quadratic) comparison to find a...
Support lock implication in allocation
Make the module abstractly handling lock allocationhonor the additional restrictions caused by lockimplications.
Add a convenience function to free all locks of an owner
While freeing all locks of a single owner can easily be definedout of listLocks and updateLocks, it is worth having this functionin its own right. For example, it will be needed when the death of...
Make GanetiLocks an instance of Lock
Add a type class describing types that can serve as locks
Besides the lock order and Show, an additional datum is needed in orderto describe lock inclusion, used, e.g., for group locks. So add a typeclass describing this property.
Add missing documentation to exported function listLocks
The module Ganeti.Locking.Allocation exports the function listLocks,hence a documentation string is required. Add it.
Conflicts: NEWS: take both additions configure.ac: ignore suffix bump...
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Constant for instance communication network mode
Create a new constant to hold the instance communication network modeas this constant will be necessary during the QA, and update thegeneral documentation about the constants related to the instancecommunication mechanism....
Add '-c | --instance-communication' flag to instance modify
Fix param name to conform to convention of optional params
The opcode parameters can be optional and parameters that are optionalhave their names prefixed by 'pOpt'. This patch fixes with parameterto conform to this convention.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>...
Fix whitespace and typos in comments
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Disable deprecation warnings in PyRPC
Since on Debian Squeeze we have TH version 2.4, we need to use 'report'instead of 'reportWarning/reportError'. However, 'report' is deprecatedin later versions, therefore we need to disable the warning.
The warning doesn't manifest on Debian Squeeze or Wheezy, but causes...
Make the lock allocation part of WConfD's state
As WConfD is supposed to be the authoritative source forconfiguration and locks, it needs to have access to the lockstate as well.