Make PrintGenericInfo handle tuples better
The PrintGenericInfo function in cli.py did not handle tuples ascontainers of items, making it impossible for these to be deserializedautomatically when a YAML parser is used. This patch adds separatehandling of tuples, including inlining them for readability when...
Refactor move-instance QA test
The move-instance QA test will have to be changed in the followingpatches to allow testing opportunistic locking.
This patch retains the same functionality as before, but allows aniallocator to be used, and splits the move back and forth into two...
Make gnt-debug delay interruptible
The gnt-debug delay command could be useful as a means of acquiringlocks for testing purposes. In practice, to be useful it should beinterruptible, otherwise we risk race conditions or long delays.
This patch follows the examples of the move-instance command and the...
Add new log type
Following the example of the job-queue test, the delay also defines alog message type of its own.
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Add the interruptible option to gnt-debug delay
This patch allows the opcode option to be used through the gnt-debugclient.
Add the interruptible option to the delay opcode
To prepare for the use of sockets to interrupt a running delay opcode,this patch adds an option to the opcode for doing so.
Factor Unix domain socket creation into helper class
As the delay class will also have to start using domain sockets,extract the functionality into a helper class.
Fix minor accidental concatenation
Fix failing tests
The previously introduced changes caused some tests to fail. This patchalters the tests to expect different values, while still ensuringcorrect behavior.
Handle incorrect duration more elegantly
The previous version of the LUTestDelay opcode relied on the utilityfunction complaining about the negative duration. As this function hasbeen removed for now, do the check ourselves, and issue a moreappropriate exception....
Make gnt-debug delay command run in parallel
The gnt-debug delay command executes the delay first on the master, andonly then on all the other nodes, causing a significant delay. Thispatch makes the command treat the master as it would all other nodes....
Extend RAPI client constant unit test with error codes
The RAPI client is meant to be independent from the wider Ganeticodebase, but it still needs to use Ganeti constants. To enforce this,a unit test is used. This patch extends the unit test with error code...
Fix typo in RAPI client utility
Add variation to delays between creation attempts
Attempting to acquire node locks with a certain fixed delay could leadto unwanted synchronization with other processes competing for the samelocks. This patch adds some random variation to the attempts....
Implement opportunistic locking in the move-instance tool
This patch allows opportunistic instance creations to be attempted,with a delay between them giving nodes the opportunity to becomeavailable. After the given number of opportunistic tries has been...
Add options for opportunistic creation during moves
To start off the introduction of oppportunistic locking during instancecreation, this patch adds the options allowing the locking to beinvoked.
Remove duplicated '_CheckOSVariant'
It seems '_CheckOSVariant' was moved from 'ganeti.cmdlib.instance' to'ganeti.cmdlib.instance_utils' but the source was never deleted. Thispatch deletes the source copy if this function.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>...
Use 'ArrayObject' in Ganeti.RPC instead of manual encoding
Before most of the calls were implemented manually, ignoring anygenerated code for maniupulating JSON. Using 'ArrayObject' we canautomate the process.
Signed-off-by: Petr Pudlak <pudlak@google.com>...
Test the instance 'ArrayObject Disk'
.. which is one of the few ones that use extra fields.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add a helper function for testing 'ArrayObject' instances
It checks if 'fromJSArray . toJSArray' is equivalent to 'Ok'.
Move 'DictObject' and 'ArrayObject' to Ganeti.JSON
The type classes are more closely related to JSON and for using themit should not be necessary to import Ganeti.THH.
Add a typeclass for type serializable as a JSON array
This is in particular needed for RPC requests which are serialized thisway and therefore using `showJSON` on the generated objects doesn'twork.
The fiels are serialized in the order they're declared, together with...
Document 'fieldExtraKeys' in 'Field'
Remove toDist... functions completely
Since thre already is `DictObject` class with `toDict`, there is no needfor them, just polluting the name space.
Enhance DictObject and refactor loading/saving TH functions
Since there is already `DictObject` type class which could share codewith generated instances of `showJSON`, the generated code is split intotwo parts: One that converts between `[(String, JSValue)]` and the data...
Write lock state to disk after updates
Make use of asynchronous worker to save to log status after everyupdate. The answer to the query is only returned, once the stateis saved on disk. Note that replication of the lock status is notnecessary, as all jobs will be gone if the master node goes away....
Provide an asynchronous task for writing lock status
It will be used to persist the state of the lock allocation ondisk, allowing locks to survive reboots of WConfD.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Verify the intervall property of GanetiLocks
According to the specification in our design, if A is a grouplock and B belogs to this group, then all locks between A and Bin the lock order also must belong to A. Verify this.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Verify implication restriction on locks
As specified in our design, if A is a group lock and Bbelongs to A, then A must be earlier in the lock orderthan B. Verify this property.
Update Arbitrary GanetiLocks
With the extension of Ganeti.Locking.Locks.GanetiLocks to thefull lock hierarchy, also extend the instantiation of Arbitrary.
Add Ganeti Lock Hierarchy
Make Ganeti.Locking.Locks.GanetiLocks present the full Ganetilock hierarchy. This is in accordance with the current Pythonimplementation.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>...
Serialize locks as JSON strings
...as the current Python implementation does as well.
Export opportunistic lock union through WConfd
Make WConfD provide a function to opportunistically allocatelocks.
Support name generation for pairs
Add support for naming of python variables if the typeis a pair.
Note: on older versions of haskell, Q is not an instance ofApplicative, hence we use the do-syntax here.
Export lock intersection through WConfD
Export the functionality to restrict the owned locks to a given set.
Export freeLocks through WConfD
Export the functionality to free all locks of a givenowner.
Add lock modification without result to WConfdMonad
Enrich the structure of WConfdMonad by a conveniencewrapper around modifyLockAllocation for the case thatthe modification always succeeds and does not produceany result.
Export updateLocks through WConfD
Export the functionality to update the locks of a givenowner through WConfD.
Add serialization for LockRequest GanetiLocks
Add a module describing how abstract requests, like lock requests arepassed over the wire to WConfD.
Add listlocks to gnt-debug wconfd
So that wconfd's locking can be debugged directly.
Export listing of locks
Make WconfD answer requests about the locks of a specificowner.
Support reading the lock allocation in WConfdMonad
As opposed to the atomic modify-and-read provided my modifyLockAllocation,no storing is necessary here.
Remove unused import in rpc/transport.py
.. which got there by mistake.
Merge branch 'stable-2.11' into master
Retry luxi/wconfd RPC calls if the connection is closed
Since the daemon can decide to close a client connection after atimeout, the client needs to be able to automatically reconnect.
This patch introduces this functionality into the RPC client:If an attempt to send data fails on 'Broken pipe', it's retried one more...
Fix generic uncurrying
in the case of n>1, the n-ary uncurry function is of the form\f (x_1, ..., x_n) -> ... rather than \f x_1 ... x_n -> ...
Add --mac-prefix to gnt-cluster modify manpage
Update the manpage for mac-prefix modification in 'gnt-cluster modify'command.
Signed-off-by: Dimitris Bliablias <bl.dimitris@gmail.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Add unittest for gnt-cluster modify --mac-prefix
Add unittest for cluster mac-prefix modification support.
Allow cluster mac prefix modification
Extend LUClusterSetParams to allow the modification of the clustermac-prefix setting in 'gnt-cluster modify' command.
This fixes part of issue 239.
Show mac prefix setting in gnt-cluster info
Include mac-prefix setting in the output of 'gnt-cluster info'command.
Setting correct permissions of client cert (split-user)
This patch makes sure that the client certificate getsthe right permissions and owner when created. Additionallyit enhances the 'ensure_dirs' script to correct thepermissions in case they are broken for whatever reason....
Add luxid group to 'users-setup'
The creation of the luxid group was missing in the'users-setup' script. Ganeti will refuse to startif this group does not exist on a node when usingsplit-user mode.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Jose Lopes <jabolopes@google.com>
Add documentation for "gnt-debug wconfd"
The missing part caused a doclint test to fail.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Add a command to gnt-debug to test various aspects of wconfd
For debugging purpose, support direct communication to WConfD fromthe commandline for some of its commands. For the time being, supportthe echo command.
Add some whitespace to fix formatting
Some error messages were lacking some spaces between linesto make it more readable.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Consider old client cert only when available
This fixes a bug which occurred only after upgradingfrom 2.10 to 2.11. During the cluster renew-cryptooperation, Ganeti tries to include the old certificatein the candidate map while it is providing newcertificates. This failed when there was no certificate...
Fix return of 'Validate'
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Locks are owned by job id file path pairs
...according to our updated design.
Remove the client state from WConfD monad
As we decided to keep our protocols stateless, the clientstate is no longer needed.
design: keep WConfD's protocols stateless
To simplify the setup, keep all protocols of WConfD stateless. This impliesthat the job id has to be sent with every resource request. Also clarify thatthe owner of a resource is the pair of the job id and the file for job death...
Have SubmitManyJobs add entries to the reason trail
Not only SubmitJobToDrainedQueue (and therefore SubmitJob) but alsoSubmitManyJobs has to add "gnt:opcode:*" entries to the reason trail.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Fix the dependencies of Makefile.ghc
Include only the programs that were configured to be built, and let GHCfigure out the required Haskell sources by itself.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Include mond and confd only when configured
Previously they were included no matter of what the configuration was,failing to compile if some of the dependencies were missing.
Add reason for job pickup to the trail
Add a new entry in the reason trail when a job is picked up by MasterD from thehard drive, after LuxiD put it there.
Note that the signature of NameToReasonSrc is changed in an incompatible way,although it's a public method because in this commit we also change its only...
Add reason trail pickup constant
Add a constant for the reason trail, representing the pickup of a job from thedisk.
Also, refactor a bit the other constants so that part of the definition can beshared in a hierarichical fashion.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Make the AddReason method public
It will need to be accessed from outside the class too in one of the nextcommits.
Have LuxiD add the "gnt:opcode" reason trail entry
The entry used to be added in jqueue.py, but after switching the queuemanagement from masterd to luxyd it had been lost. Now, make LuxiD responsiblefor adding it.
Add function for extending the reason trail in Luxid
The function will be used by the next commit.
Also, remove a few trailing whitespaces lying around the file.
Update INSTALL wrt missing dependencies
This patch updates the quick installation guide withwith respect to some missing dependencies. Besides,it rephrases some instructions to make them more clear.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
hsqueeze also partially implemented in 2.11
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Let config.py use WConfd for reading/writing the config
Currently it only relays the reads/writes to the file to WConfd,everything else yet remains in config.py.
Also if the 'ConfigWriter' is opened in "offline" mode (like inbootstrap.py), it doesn't use WConfd and resorts to the original...
Start WConfd temporarily during master failover
.. in order to update the configuration and distribute ssconf, beforestarting the daemons by the scripts.
Include WConfd in the list of daemons started on a master
.. and make it the first one so that masterd can use it when it starts.
Add missing constant for the name of WConfd
The constant is used when calling the daemon startup/shutdown scripts.
Add the --force-node option to WConfd
.. and fix the superfluous options list in ganeti-wconfd.hs
Add an option for disabling node checks at daemon startup
This is needed for WConfd, because during failover it needs to betemporarily started in order to update the configuration on the selectedmaster candidate.
Convert int to float when checking config. consistency
When reading the configuration file from RPC JSON, values without afloating point are parsed as 'int', not as 'float', and later theconsistency check fails.
This patch adds an automatic conversion from 'int' to 'float' during...
While at it, fix the order of imports in OpCodes.hs
.. so that Ganeti imports are below library imports and orderedalphabetically.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Add a function for generating OpCode reason src. names
The function convert the opcode name to lowercase with underscores,strips the 'Op' prefix and prepends Constants.opcodeReasonSrcOpcode.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>...
Add a TH function for lower-cased stripped opcode names
The function strips the 'Op' prefix from a constructor name and convertsit to lower-case with underscores.
Generalize genConstrToStr to custom monadic functions
This will allow compile-time checks for constructor names.
Prefer unless over when (not...)
...and by doing so, please lint.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
At startup, make WConfD read the lock allocation from disk
On startup, make wconfd read a the state of the lock allocation from disk,instead of starting from an empty allocation.
Note that, as opposed to the configuration, no finding a savedlog configuration is not an error; e.g., the node might be master...
Provide path to persist locks
As we aim to handle jobs as separate processes with lockshandled by WConfD, we need to persist the locks to ensurethey survive a restart of that deamon. Saving only on masterdaemon is enough, however.
Add a function to load lock allocation from disk
Lock allocation are stored on disk as their JSON representation.
Verify serialization/deserialization consistency for Locks
As we made Ganeti.Locking.Locks an instance of JSON, we better verifythat we can readJSON what we showJSON.
Make Locks a JSON instance
As requests to acquire or release locks come viainter-process communication, we need a serializationof the locks data type.
All locks are serialised as JSON objects, where thekey "type" describes the type of lock, and the remaining...
Add test for serialization of LockAllocation
Verify readJSON . showJSON = Ok for LockAllocations.
Make LockAllocation an instance of JSON
In this way, we can serialize a lock state to disk, so thatlocks can survive a restart of WConfD.
Merge branch 'origin/stable-2.10' into stable-2.11
Signed-off-by: Hrvoje Ribicic <riba@google.com>...
Add query support for locks to luxid
While requests only get forwarded, it still helps toget luxid feature-complete with respect to master.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>Cherry-picked from commit a6e406ce376453e90e598c7be68809d6a7bd7d41...
Provide fields for lock queries
For luxid to be feature-complete with respect to masterd, italso needs to answer requests about locks. This includes knowingthe fields available for locks.
Export reading and writing the configuration using RPC
Note that currently these functions lack any kind of locking andconcurrent calls to 'writeConfig' will result in lost changes in theconfiguration.
Add a module for functions specific to config. management
.. and include ConfigData in WConfd's ConfigState.
Currently it includes functions for asynchronous saving theconfiguration and reading and (over)writing it in the daemon state.
Load the configuration from the file at daemon startup and configure an...
Add more general conf. handling functions to WConfd's Monad
Add functions for reading and writing the configuration. Writingtriggers the asynchronous worker which saves the configuration into themaster file. The actual action that saves the configuration is passed as...
Add functions for saving the configuration to Ganeti.Config
.. so that it can be not only read, but also written to the masterconfiguration file.
Add a module for creating and using asynchronous workers
Each such worker has its own background thread and performs a giventask when triggered. Triggers arriving when the worker is processing areaccumulated and processed together in a subsequent worker action....
A separate module for functions for atomic file operations
Utils is getting too big, so better split this new set of functions intoa separate sub-module. This also allows us to use ResultG there.
Add a function for lifting ResultT from a MonadBase
Often we have `ResultT e IO a` as the return type of our primitivefunctions and we need to lift them into a monad stack that is also basedon IO. This function accomplishes this by lifting a ResultT in a base...