'LUClusterSetParams' creates the instance communication net
Extend 'LUClusterSetParams' to create the user-supplied instancecommunication network in case this network does not exist. Note thatif the user-supplied network already exists, nothing needs to be done...
Add 'instance_communication_network' to 'OpClusterSetParams'
... and update the unit tests.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add 'instance_communication_parameter' to 'Cluster'
Fix indentation
Merge branch 'stable-2.11' into master
Remove the HTOOLS configuration variable
.. and update the code that uses it.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Conflicts: lib/client/gnt_node.py: trivial src/Ganeti/Query/Query.hs: import ALL the functions
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Consider job-IDs queried for twice only once
As reading jobs from disk is an expensive operation, when queryingfor jobs, we optimize by considering which values the job-id is askedfor in the filter. As any reasonable person would not add the sameclause twice in an Or-clause, the implicit assumption was that the...
Merge branch 'stable-2.10' into stable-2.11
Merge branch 'stable-2.9' into stable-2.10
Fix 'JobIdListOnly' type from 'List' to 'Map'
Allow classic queries to use either names or UUIDs
When UUIDs are used in CLI commands, such addressing of objects failsor succeeds inconsistently across object types. Worse yet, some callsdo not fail, but simply return no result. This is due to the way the...
luxid: give stern warnings about debug mode
Luxid as it is can leak private and secret parameters by loggingall requests as they arrive, before any preprocessing is done.
Warn the user stern warnings about this.
Signed-off-by: Santi Raffa <rsanti@google.com>...
OpCodes: modify InstanceReinstall for private, secret params
Modify InstanceReinstall to accept and process private and secretparameters.
Signed-off-by: Santi Raffa <rsanti@google.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
OpCodes: modify InstanceCreate for private, secret params
Modify InstanceCreate to accept process private and secret parameters.
OpCodes: modify ClusterSetParams for private parameters
Modify ClusterSetParams to accept and process private parameters.
OpCodes: modify InstanceSetParams for private parameters
Modify InstanceSetParams to accept and process private parameters.
Add private OS parameters to cluster and instance conf
This updates objects, constructors and mocks for Instance and Clusterobjects in Python and Haskell.
Add Private types to Python, Haskell
This commit adds the private containers to Python and Haskell.
serializer: emit and encode Private values
For inbound data the simplest, safest thing to do is to traverse allJSON right after encoding and search for private parameters by key.
This ensures that all consumers of this data get Private valuestransparently and consistently; the serializing methods don't have to...
Remove wildcard luxi operation matching in luxid
In that way, we explicitly name the operations that are nothandled by luxid and explain the reason. In particular, wecan be sure that newly added luxid operations won't be forgottenin luxid.
Implement QueryExports in luxid
...by handling as a classical query, using that queries forexport are already implemented. Note that QueryExport is slightlydifferent from other Query* Luxi requests, in that the fields arenot passed with the request, but have a fixed value....
Add CLI and OpCode instance communication option
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Generate TAP names ('gnt.com.%d') for instance comm NICs
Add metadata daemon
... including, user, group, daemonizing code with command lineoptions, integration with the Snap HTTP server, and logic.
Change return type of internal rmJob
...to also provide the job itself. In this way, the function canalso be used for tasks that require temporarily removing a jobfrom the queue.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
When enqueuing new jobs, respect job ID
When adding new jobs, don't add them at the end, but at aposition that fits with their job id. In this way, we canbuild operations that require fully dequeing a job an addingit later after some modifications.
Provide a function to change the priority of a queued job
There is a separation of responsibilities here. For jobs stillin the queue, it is the responsibility of the queue (scheduler),for started jobs, the job itself has to take care of it. To avoidthe job transitioning inbetween, it is temporarily dequeued during...
Implement ChangeJobPriority in luxid
For jobs still queued, we ask the queue to change the priority,and replicate the changed job. For jobs that have already beenstarted, we have to contact the job directly, which, at the moment,means forwarding the request to masterd....
Add a function changing the priority of an opcode
This pure function follows the semantic that an opcode, includingits priority, may only be changed if the opcode is not finalized.
Add a function to change the priority of a job
...by changing the priority of the non-finished opcodes.
Add functions for manipulating errors in Result(T)
There is often need to manipulate these errors, for example to convert a Stringfrom Result into an exception. These functions make this easier.
Function 'toErrorStr' lifts 'Result' to any 'MonadError'. This is useful...
Remove FromString in favor of Error from standard libraries
They have the very same functionality, and using our own FromString onlycauses unnecessary code duplication.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>...
Add Alternative instances for GenericResult and ResultT
This allows to use Alternative specific combinators, namely `optional`.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Cherry-pick of 78209a84b0f6be27fd381ac2...
Catch exceptions in 'readConfig' in Ganeti.Config
.. and convert them into a "Result".This is accomplished by using "liftIO" from ResultT.
Catch errors in 'lockFile' in Ganeti.Utils
Opening a file can fail as well, so catch errors there too.
Assert correct usage of AndRestArguments fields
The special field andRestArguments is intended to be only usedas the last field of an object, catching all remaining keys.Add a compile-time check to verify that it is used correctly.
Fix whitespace
Add andRestArguments to IDiskParams
In this way, we cann pass through the opaque parametersrequired for disk creation and modification in the case ofexternal storage.
Add function providing the canonical andRestArguments
The field catching the remaining fields will always be of the sameshape, so add a function for this to make usage simple.
Add additional constructor AndRestArguments to OptionalType
A field of this type will capture all the remaining fieldsof an object as JSValues. Obviously, the intended use isto have precisely one such field. This mechanism will allowto pass opaque values trough, as it is, e.g., required for...
User new error handling functions in SubmitJobToDrainedQueue
This somewhat shortens and simplifies the code.
Use new error functions when querying locks
This helps to handle errors coming from the Luxi client.
Use new error handling functions for querying jobs
Since we already touched getJobIDs, and this function is already basedon ResultT, use new error functions here as well.
Update getDirJobIDs to use ResultT
Also simplify code and remove unused functions.
Improve liftIO of ResultT to catch IOErrors
Any IOErrors are now captured in ResultT's liftIO using try and converted into'Bad'. This prevents IO exceptions from leaking into ResultT code and allowsthem to be handled in a pure way inside ResultT's sum type....
Rename 'resultT' to 'toError'
.. to better correspond to its generalized type.
Generalize type signatures of functions that produce Result
This allows them to be used in any 'MonadError', in particular in monad stackscomposed using ResultT, without explicit lifting.
Add withErrorLogAt which annotates and logs errors
.. and rethrows them within a MonadError. This allows to just log errorsand let them be handled elsewhere.
Refactor instance MonadLog (ReaderT r m) to unify style
.. with MonadLog (ResultT e m) to be more point-free.
Add instance MonadLog for ResultT
This allows to log in monad stacks containing ResultT.
Replace duplicated code with readJSONWithDesc
JSON errors were annotated at other places, so use the function there aswell.
When parsing fields, annotate JSON errors with names
This allows better debugging of failed JSON conversions.
Add a function that enhances readJSON error messages
If an error occurs, the error message is annotated with a given textualdescription of the parsed type and optionally also with the input data.
Make safeRenameFile create dirs with defined permissions
If, and only if, safeRenameFile creates a new directory, make sureit has well defined permissions. While there, also optimize for thecommon case. The main use of safeRenameFile is archiving jobs. As...
Add constant for subdir permissions within the job queue
When archiving jobs, new directories have to be created, asjobs are archived in groups of 10000. Add a constant describingthe permissions of these newly created directories.
Note that, due to the type, the constant cannot be part...
Add utility to fix permissions
Especially when creating new directories, we need to make sureownership and permissions are set correctly. Provide a functionto do so.
Add data type describing permissions and possibly owners
When creating new files, and, more importantly, new directoriesit is relevant to set permissions, and possibly owners, correctly.Provide a type specifying the target configuration.
Enable network tags in Haskell code
Prior to the creation of the 2.10 branch, network tags were broken, andthe Haskell code introduced there mistakenly accepted this as thedesired functionality. This patch fixes this in a very simple way.
Signed-off-by: Hrvoje Ribicic <riba@google.com>...
Add 'provider' to IDiskParams
IDISK_PROVIDER was included in python's IDISK_PARAMS, so itshould also be included in the Haskell code.
Now that luxid creates and enqueues jobs, without this patch theExtStorage interface is broken as the user can not pass the disk...
Disabling client certificate usage
This patch temporarily disables the usage of the clientSSL certificates. The handling of RPC connections had aconceptional flaw, because the certificates lack a propersignature. For this, Ganeti needs to implement a CA,...
Implement auto-archiving of jobs
As luxid is taking over the handling of the job queue, italso needs to handle the automated archiving of jobs. Herewe replicate the semantics of the current python implementationof archiving as many jobs older than the given time as possible,...
Add a utility function to try archiving jobs
Provide a function that walks through a list of job ids andarchives them if appropriate. Abort that process if a giventimeout is reached.
Support computation on Timestamp
As timestamps are also used to determine if an event is sufficientlylong in the past (e.g., on archiving jobs), support adding a timeinterval to a Timestamp.
Add constructor function for Timestamp
Provide means to get Ganeti's internal timestampsfrom standard clock time.
Add a predicate on Jobs on whether it can be archived
Jobs usually are archived a given time after they have finished.For finalized jobs without end-time, the start-time is taken enlieu. This function provides the pure predicate for this decision.
Make ArchiveJob in luxid create the archive, if necessary
As jobs are archived in groups of 10000, creating new subdirectoriesof the archive might be necessary when archiving a job. Use afunction that takes care of this.
Provide a safe version of rename
...that also creates the target directory, if needed.
Fix expectation for the return value of jobqueue_rename
On sucess, jobqueue_rename returns a list containing onenull per change request.
fix off-by-one error in indentation
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Implement ArchiveJob queries in luxid
With luxid taking over the tasks of masterd, archivingjobs also belongs to its responsibilities. As archivinga job affects the global state of the job queue, synchroniseover the queue lock.
Add RPC call jobqueue_rename
Archiving jobs is also replicated to all master candidates.Therefore luxid needs to be aware of this RPC call.
Make use of fieldListToFieldMap
...to avoid duplicating that code all over Ganeti.Query.
Provide a utility function to map FieldList to FieldMap
As the same construction is used in several places, it is betterto have it factored out as a named function.
Add query support for locks to luxid
While requests only get forwarded, it still helps toget luxid feature-complete with respect to master.
Provide fields for lock queries
For luxid to be feature-complete with respect to masterd, italso needs to answer requests about locks. This includes knowingthe fields available for locks.
Remove dead Ganeti.Query.Job.loadRuntimeData
This function was exported from the module, but actually neverused anywhere in the code base. So clean it up.
luxid: fix detection of master node in node query
Ganeti.Config.getNodeRole would rely on clusterMasterNode returning themaster node name, however clusterMasterNode returns the master node'sUUID. We fix this and a similar issue in Ganeti.Query.Node.nodeFields....
When updating job queue, support virtual paths
When replicating parts of the job queue, allow for virtualpaths in the RPC call. In this way, replication will alsowork correctly in a vcluster setup. Note that makeVirtualPathlives in IO, and hence cannot be part of the pure encoding...
Add a module to support virtual clusters
Virtual clusters are an efficient way to test how Ganeti behaveson a large cluster without requiring a large number of machines.Now that more tasks like job replication are done by luxid, providethat functionality in Haskell as well....
Move vcluster-related constants to Constants.hs
...as, in that way, they will also be available in Haskell,where job replication happens as well.
Update set_watcher_pause to use ClockTime instead of Double
This only affects the internal representation in the Haskell part.
Clean up luxidMaxRunningJobs
Now that the number of jobs maximally running in parallel isa run-time option, this magic constant is not needed any more.
Make the scheduler use the max_running_jobs config parameter
Use the run-time configuration to decide on the number of jobsscheduled for execution instead of using a hard-coded constant.
Make configuration available to the scheduler
In this way, scheduling decisions can depend on the configurationof the cluster. At the moment, this is only the maximal numberjobs to be run in parallel, but in the future this will also includejob filters....
Make max_running_jobs queryable
As we have introduced a new cluster parameter, it shouldbe also visible when querying about the cluster configuration.
Add opcode parameter for the maximal number of running jobs
This parameter of OpClusterSetParams will allow to set themaximal number of jobs to be run simultaneously.
Add parameter max_running_jobs to the cluster configuration
This cluster-wide parameter will determine how many non-finalized jobs maximallyshould be in a not queued state at the same time.
Use ClockTime instead of Double in fields in Objects.hs
This affects "mtime" and "ctime" fields in all data types.
This also forces explicit declaration of how the fields are serializedin Query.
A function for creating time fields that serialize as Double
The function creates fields with custom read/show methods based onTimeAsDoubleJSON, with the default value of 0 seconds since the epoch.