Add Haskell and Python modules for running jobs as processes
They will be used by Luxi daemon to spawn jobs as separate processes.
The communication protocol between the Luxi daemon and a spawned processis described in the documentation of module Ganeti.Query.Exec....
Add an utility function for writing and replicating a job
Use the function where appropriate.
Also handling of CancelJob is slightly refactored to use ResultT, whichis used by the new function.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add a livelock file for the Luxi daemon
The file is initialized and kept within JQStatus.It is temporarily assigned to jobs spawned by Luxi until they createtheir own livelock files.
Move `isDead` from DeathDetection to Utils/Livelock
.. as it has nothing special to do with WConfd and fits the new modulebetter.
Add a module for livelock related functions
Currently it exports a function for creating livelock files.
Add functions for computing the full path of livelock files
.. so that Haskell code can create them at the proper place.
Allow closing a RPC client, keeping its file descriptors
The purpose is to keep the communication channel open, while replacing a'Client' with something else.
Separate client and server config for Luxi communication
The daemon identity is only required for server connections to set theaccess mode to its socket appropriately. For client connections it'snot needed and requiring it prevents creating standalone clients, for...
Extend 'lockFile' to return the file descriptor
.. of the locked file so that it can be closed later, if needed.
Allow creation if bi-directional pair of Luxi-like clients
This allows a process and its forked child to communicate with eachother using our standard infrastructure.
Separate read and write handle in the Luxi Client data type
This is required for inter-process pipes, which are fully supported onlyas uni-directional.
Report non-existent jobs as such
When queried to WaitForJobChange of an non-existent job,report this as an error.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
fix typo
Clean up from LockAllocation what is no longer used
With the change from LockAllocations to LockWaitings, severalmanipulation operations had to be implemented for LockWaitingsand became unused in LockAllocation. Remove these functions thatare no longer used....
Use a LockWaiting structure instead of a LockAllocation
In this way, we will be able to support in WConfD waiting for locksto become available instead of having to poll for them.
Support opportunistically allocating locks
Again, this just wraps around updateLocks, sequentially tryingto obtain all the locks mentioned.
Support intersecting locks
...in waiting structures. This is just a convenience wrapperaround freeLocksPredicate.
Support downgrade and freeing locks
Add convenience functions for downgrading and freeinglocks in a waiting structure. As these functions areguaranteed to always succeed, they will also drop anypending request of the owner.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Do not notify the current requester
The current implementation of lock waiting yields as notificationset the list of all owners whose requests could be fulfilled. Thisincludes the initiating request. While technically correct, theoriginal requester gets the answer of the request and hence does not...
Fix a race in rescheduling jobs
When handling the queue, in particular at analyzing job dependencies,we assume that all non-finalized jobs are present in the Queue datastructure. When rescheduling jobs we move them from the running partof the queue to the scheduled part again. In order to comply with the...
Instance JSON LockWaiting
...so that it can be used by WConfD who needs to beable to persist all its data.
Instance JSON LockRequest
...as pending requests need to be serialized when serializinglock waiting structures.
Add a function to get a LockWaiting from an extRepr
Add a function to obtain some lock waiting structure from agiven extensional representation.
Provide an extensional representation of LockWaiting
The internal representation of the lock waiting structure containssome arbitrariness---pending requests are arbitrarily keyed by oneof the lock owners that blocks them. Therefore, LockWaiting is notan instance of Eq. To allow some meaningful form of comparison...
Export the set of pending requests
...from a lock waiting structure. In this way, all the datadescribing the behavior can be inspected.
Schedule on jobs where all job dependencies are finished
Jobs may depend on other jobs in the sense that they may only be startedonce a given job is finalized. For a job process, however, it is hard todetermine if the status of a different job without a significant overhead....
Add function to obtain job dependencies
Add a function that computes the list of job ids a job dependson. This will allow to schedule only those jobs for execution,where all jobs it depends on have been finalized.
Add function to obtain job id from a dependency
Add a function that extracts the job id, if given in absoluteform, from a job dependency.
Add process id to lock-owner description
...so that we can notify owners when their pending request got granted.
Add a function to release all resources of an owner
Add a convenience function---definable from the exportedinterface---to clean up all the resources of an owner. Thiswill be needed, e.g., if a job dies.
Add a function to remove a pending request
Add a function that allows an owner to remove a pending requestit might have. Besides useful for testing, this will also beneeded to clean up the resources for dead owners.
Export the set of pending owners
...as the question of who has pending requests is an importpart of the data structure.
Export the underlying allocation from a waiting structure
...so that all kind of diagnostic tasks can be carried out withoutduplicating the functionality on the waiting structure. Note thatwe still do not export the constructor so that we can change the...
Add functions to update locks in a waiting structure
Add functions to update the lock allocation structure and alsocompute all the consequences of the update, if it succeeds. Optionally,requests that cannot be fulfilled immediately, it can be added tothe pendig requests....
Add data structure for lock waiting
To avoid jobs polling on locks, and also to ensure that themost important of the waiting jobs obtains a lock, add adata structure to keep track of who is waiting for which locks.
Derive Ord for LockRequests
Add zeroing timeout params to opcode
This patch adds two parameters controlling the zeroing timeout - onethat is fixed and another that depends on the amount of data (size ofdisks) to zero.
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Add file info RPC call
Due to the need to determine the size of the temporary disk used forthe VM helper, an RPC call capable of supplying this information hasbeen added. It can be extended as needed to supply additionalinformation provided by Python's os.stat....
Add the zeroing-image option
This patch adds the zeroing-image option to gnt-cluster and theOpBackupExport params. The many changes are all minor, yet necessary.
Add the zero-free-space option
Although unused for now, add the zero-free-space option to the backupexport opcode, and add a rudimentary check for whether it should beused.
Serve OS scripts from metadata daemon
This patch makes the metadata daemon web server serve the OS scriptsvia the instance communication mechanism. This is achieved byretrieving the "os" field from the instance parameters and serving thefile directly from the 'Constants.osSearchPath', typically...
Move processing OS params with visibility to web server
This will allow the metadata daemon to keep a more general instanceconfiguration and thus serve more general requests, such as, servinginformation related to the OS scripts.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>...
Move metad configuration related code to a separate module
... as an intermediate step before moving the responsbility ofcomputing the final OS parameter configuration from the configurationserver to the web server. This will allow the metadata daemon to...
Switch to ClientType as identifier
...instead of Either String JobId.
Instance JSON ClientType
Make ClientType a JSON instance, in the way lock owners arealready encoded in JSON on the Python side.
Add a type for the entity owning a lock
While it is unlikely that lock owners will ever be represented bysomething different than a job id or, exceptionally, a string, thistype will still improves type checking. Additionally, it will allowus to make it a separate instance of JSON and thus simplify the...
Make LuxiD query WConfD for locks
Since WConfD is now the authoritative source for locks, make LuxiDquery this daemon for lock information rather than the master daemon.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Make locks field use live data
So far, the description of the locks fields was made under the assumptionthat lock queries wouldn't be answered by Luxid anyway, and hence it wasenough to parse such requests. However, now luxid will answer these queriesafter getting a snapshot of the locks status from wconfd. Hence make the fields...
Export lock name
...as locks are queried by name.
Expose listAllLocksOwners in WConfD
...so that LuxiD can ask for the lock status to answerlock queries.
Add a function to provide all the full lock state
This function will be used to provide the information requiredto answer lock queries.
Expose listAllLocks on WConfD
Make wconfd provide information about the list of all locks currentlyin use.
Add a function to obtain all active locks
Add a function that extracts from a LockAllocation the listof all locks that are in use (directly or indirectly).
Fix lint errors after moving Luxi code to UDSServer
Remove unused imports and update the 'buildCall' call in tests.
Add a WConfd RPC Client
The client is generated automatically by Ganeti.THH.HsRPC.
It also includes helper functions for creating a client object.
Add a module for creating Haskell-to-Haskell RPC clients
It uses TH to create client functions with the corresponding signaturefrom a given list of server functions.
Add a generic TH currying function for arbitrary arity
.. and also a function for extracting a type argument from a type.
Move buildCall and validateResult from Luxi to UDSServer
.. to that they're available for other Luxi-like RPC clients as well.
While at it, rename validateResult to parseResponse to be consistentwith the names of the other functions.
Signed-off-by: Petr Pudlak <pudlak@google.com>...
Serve OS package from metadata daemon
... and fix an error message.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Fix metad port from 8080 to 80
Add metadata daemon server entry point
... which starts the configuration and web servers.
Modify metadata daemon web server to serve OS parameters
Extend metadata web server to serve the OS parameters from the sharedMVar containing the instances' configurations, indexed by their IPaddress.
Add metadata daemon configuration server
The configuration server listens on a Unix socket for connections fromthe node daemon. The node daemon sends the instance parameters to theconfiguration server so they can be served through the metadata daemonweb server to the instances which have the communication mechanism...
Add metadata daemon types
The metadata daemon will be responsible for receiving the instanceparameters from the node daemon and keeping them in memory. The typeof the instance parameters in defined in a separate module because itwill be shared between the web server and the configuration server....
Move metadata daemon to its own directory
Currently, the metadata daemon is mainly a web server. However, a newconfiguration server will be introduced along with other metadatarelated modules. Therefore, this patch moves the current metadatadaemon web server to its own directory. This module is also renamed...
Log asynchronous WConfd tasks and include the serial number
This makes debugging more informative and makes more clear howconfiguration distribution gets aggregated.
Enable logging of thread ID in Haskell daemons
Identify lock owners by either string or job id
Most requests asking for resources are jobs. However, in exceptionalcases, other requests (like currently requests to masterd requiring theconfiguration) need to ask for resources. They identify themselves by...
Merge branch 'stable-2.11' into master
Merge branch 'stable-2.10' into stable-2.11
Merge branch 'stable-2.9' into stable-2.10
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>...
Fix failing gnt-node list-drbd command
The UUID/name switch avoided this particular bit of code, and as aresult the list-drbd command failed as it tried to compare UUIDs andnames. This patch fixes the retrieval, converts the newly returnedUUIDs to names, and modifies the QA to the results and not only the...
Add lock checks to readConfig and writeConfig in WConfd
Allow reading the configuration only if the client has acquiredConfigLock in shared or exclusive mode, and allow writing it only if theclient has acquired ConfigLock in exclusive mode.
Add RPC calls to WConfd to lock/unlock ConfigLock
This allows the clients to protect configuration operations, as requiredby the current implementation of Python 'ConfigWriter'.
Add a function for determining if a client holds a lock
This convenience function examines 'listLocks' for a particular ownerand checks, if it holds a given lock at a given ownership level (orhigher). Therefore checking for a shared locks returns 'True' if the...
After each lock operation, log the current lock status
.. using the DEBUG channel.
Add a transitional configuration lock
The lock is used for protecting concurrent operations on theconfiguration on the Python side.
The lock must be last wrt to the lock ordering so that it can be alwaysobtained regardless of other locks held by a client. During operations...
Include the master in the list of MCs in SSConf
This was an unnoticed regression when SSConf has been refactored intoWConfd.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add a function to return all MCs including the master
The original 'getMasterCandidates' returns only the true candidates,without the master itself.
Also consider filter fields for deciding if using live data
If the query fields don't require live data, we use the shortcutand don't request live data. However, we cannot take this shortcutif the fields the filter depends on requires live data.
kvm: Add migration capabilities as an hvparam
Latest QEMU versions support various migration capabilities. Eachcan be enabled/disabled with 'migrate_set_capability' monitorcommand.
Version 1.7.0 defines x-rdma-pin-all, auto-converge, zero-blocks,and xbzrle migration capabilities....
Add a thread ID to the WConfd client id
This allows to distinguish threads that don't have a job id, which isneeded for answering queries.
Since Python thread IDs aren't guaranteed to be unique, in future it'dbe preferable to use a different, unique identifier....
Create a separate data type for WConfd client identity
.. instead of using a pair.
Add downgrading locks by level to WConfd
Export a function that downgrades all locks of an owner of agiven level to own them only shared.
Add convenience function to downgrade locks
Add a convenience function that downgrades all locks with a givenproperty to be held only at shared level.
Fix typo in lock levels
Helper function that downloads an image and dumps it to disk
Support releasing locks by level
Add a function to WConfD allowing to release locks byGaneti lock level.
Add a convenience function to release locks by property
While definable from listLocks and updateLocks, it is convenientto be able to all locks with a given property. This will, e.g.,allow to release all locks at a given level---as the mcpu doesroutinely....
BGL is not a set lock
Despite what the name suggests, the Big Ganeti Lock does not act as aset of all locks. It is just a lock as any other lock from the pointof view of the Ganeti lock hierarchy.
It's special meaning comes from the fact that all jobs, except for...
For each Ganeti Lock provide its level
...so set a job can request operations on all its locks of a given level.
Add network locks to Haskell
Ganeti also has a lock level for networks, however this level was forgottenwhen the Ganeti lock hierarchy was implemented in Haskell.
Fix allocation order of locks
The correct allocation order for node-related Ganeti locks isallocation, group, the node itself, resources.
Allow transitions from sets to elements
Owning a set-lock has the same effect as owning each member. So askingfor a member while already holding the set lock in the same mode (sharedor exclusive) is, effectively, no asking for new locks. Therefore, do not...
Fix typo
Let WConfd distribute SSConf to nodes
.. and remove the corresponding code from lib/config.py.
Add an asynchronous task for distributing SSConf to nodes
The task computes the SSconf data and compares it to the last state thatwas successfully distributed. If it differs, issues the RPC call todistribute it. If any of the RPC calls fails, the state remains...
Add Haskell RPC call for distributing SSConf to nodes
The data is sent as a dictionary mapping file names to lists of lines.
Add a module for constructing Ssconf from the configuration
This migrates the functionality from PythonConfigWriter._UnlockedGetSsconfValues to Haskell.
Add a data type for representing the complete SSConf
.. as a map from SSKeys to lists of strings.