Gluster: use ssconf value for mountpoint directory
Gluster still does not mount anything autonomously, but this commitchanges where Gluster expects its mountpoint to be.
Signed-off-by: Santi Raffa <rsanti@google.com>Signed-off-by: Thomas Thrainer <thomasth@google.com>...
ssconf: Add Gluster mount directory
This commit adds the gluster storage directory to ssconf (withoutactually using its value just yet).
Signed-off-by: Santi Raffa <rsanti@google.com>Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Gluster: add GlusterVolume class
This commit teaches Gluster what a volume is and how to use it.
Gluster: minimal implementation
Add Gluster to Ganeti by essentially cloning the shared file behavioureverywhere in the code base.
netutils: Add ValidatePortNumber method
This method accepts a port number and checks that it is in fact valid.
FileStorage: extract file logic to a FileDeviceHelper object
This will allow code reuse for Gluster through composition, ratherthan inheritance.
FileStorage: move to filesstorage.py
Move the FileStorage class in its own file, together with its helperfunctions.
PathJoin: improve error message when given one argument
PathJoin fails with an unclear message if only one argument is passedto it. Calling PathJoin("/foo") causes this exception:
Error: path joining resulted in different prefix (/foo != /foo)
However, /foo and /foo obviously share prefixes: what this function...
ComputeLDParams: do not spell out disk templates
A large part of the complexity in this function is due to the needto translate from "template-specific" parameter names to"template-agnostic" parameter names. This logic is complex and havingcomplex code for complex logic is okay....
bdev: Fix position of DEV_MAP
This rather important dictionary from constants to classes was hidingbetween function definitions. The dict cannot go to the top of the fileas the classes haven't been defined there yet, so it's been pushedto the bottom of the file....
gnt-cluster verify: demote orphan volume error to warning
Ganeti checks for orphan volume by making sure that it knows about allvolumes on disk; any additional orphan volume, even if created by theadministrator, causes a failure in gnt-cluster verify. Given that...
For the commandline, switch to query socket by default
As luxid now understands all the requests used by the command-line tools,switch the default luxi socket for those to be the socket of luxid.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Implement fields query for instance
Support the query for the fields available for instances.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Remove the hvsGlobals from instance query fields
...to be consistent with the python implementation.
Add nic.vlans to the query fields
In commit 3293332 this was only done for the Haskell side; doso for python as well, to have both views consistent.
When interpreting [] as "all fields", sort nicely
When asked for all fields, we promise to return the list of fieldssorted according to niceSort. Keep this promise.
Take a fresh Luxi client for each failing test
Luxid is more strict with closing the connection after receivinga syntactically incorrect request, gnt-debug cannot use the sameclient for several successive tests verifying that a syntacticallyincorrect request is recognized as such....
Fix race in watchFile
As the calling of watchFile and the evaluation of the initialgetFStatSafe takes non-zero time, the file could have changedbefore inotify was set up properly. Solve this problem by anadditional check for the watched value to have changed immediately...
Use a data type when generating Python types of OpCodes
Currently they are generated only as Strings.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Refactor OpCodeDescriptor from a tuple to a data type
This greatly enhances code readability.
Also fix monadic types "Q ExpQ" [which is "Q (Q Exp)"] to "Q Exp".
Add showValueList to PyValue for proper String instances
It's the same trick ShowS uses. We add a type class function forshowing a list to PyValue and then just use it in the instance for`[a]`. This way we have the proper String instance without anyoverlapping/incoherent instances....
Rename PyValueInstances.hs to PyValue.hs
Now the file contains the type class declaration as well.
Move PyValue into PyValueInstances.hs, import it in THH.hs
This puts all PyValue code into one module, getting rid of orphaninstances.
Make the duration field optional null-serialized
The time in SetWatcherPause is optional (with Nothing meaningthat the pause should be canceled), but the serialization isnot that of a Maybe Double; instead Just values serialize asthey are and Nothing serializes to null. Fortunately, we already...
Handle QueryConfigValues
Make luxid handle the QueryConfigValues call providing certainsimple status information about the cluster.
Add a predicate for watcher pause
Add a predicate, in IO, to test whether the watcher ispaused.
Provide path to watcher pause file
Extend Path.hs to also provide the path to the file indicatingwhether watcher is paused.
Implement SetWatcherPause in luxid
Make luxid handle SetWatcherPause correctly.
Add the RPC-call set_watcher_pause
With luxid taking over responsibility for handling watcher-pause requests,it needs to know about this RPC. So have it available in Haskell as well.
The time field for SetWatcherPause is optional
A JSON null value is used to indicate that the pause should be canceled.
Generate a separate return type for the job queue update RPC
The instantiation of RPC requires a bidirectional functional dependencybetween call type and return type. Hence we cannot use Unit everywhere.
OS-redesign: change instance IP
Change the IP of the instance to make it correspond to the one used in theexample.
The .253 is chosen so that, if needed, in the future the network can be changedfrom a /24 to a /28.
Signed-off-by: Michele Tartara <mtartara@google.com>...
OS installation redesign
Add the document describing a new design for the OS installation process fornew instances.
Signed-off-by: Michele Tartara <mtartara@google.com>Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Move the generalized IO client from Luxi to UDSServer
No code is changed in this patch (except imports and qualifiers), onlymoved.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Generalize the IO client handling in Luxi
... to be usable for WConfd as well. A daemon handler is encapsulatedinto `Handler` data type, which is then passed to a generic `listener`.
The changes are done in Luxi.hs so that the differences are visible and...
Add MonadLog instance for `ReaderT r m`
This allows to use logging with the ReaderT monad transformer.
Add a MonadLog typeclass for monads that allow logging
This separates logging from IO, allowing to create unit tests in futurefor functions that use it.
Add fromJResultE and fromJVal that uses MonadError
Using MonadError is more correct than just "fail" on an arbitrarymonad, and more scalable when using monad type classes or monad stacks.
Add an Error instance for GanetiException
This allows it to be used with MonadError.
Add MonadPlus and MonadError instances for GenericResult
.. and ResultT.
While at it, generalize also the MonadPlus instance of GenericResult andadd some Functor/Applicative instances.
Generalize "validateCall" to be usable outside LUXI
Return the method (as any instance of JSON) and the arguments of a call.
Add the Unix domain socket path to the Server data type
This simplifies code for closing such a socket.
Encapsulate a server socket and its parameters
Instead of passing a bare server socket around, we pass it encapsulatedin a data type together with parameters such as read/write timeouts.
Rename getClient/Server to getLuxiClient/Server
Later they will be split into LUXI-specific and general parts.
Split Luxi.hs into LUXI-specific functions and general ones
This will allow WConfD to use the general functions without importingLuxi.hs.
Make luxid support WaitForJobChange
Make support the WaitForJobChange, waiting for a job tochange on certain monitored fields.
Add a generic function capable of watching a file
Add a method to return the new value of a function if it changes withinthe given timeout. If not, return the old value. Make use of the fact,that the function only changes, if the specified file changes on disk....
Add a safe version of getFStat
The function getFStat causes an IOError if the file to be stated doesnot exist. In some cases, however, the only thing we care about is whetherit has changed, with disappearing being a legitimate change. So add a wrapperthat catches the IOError and returns nullFStat....
Gluster: Update design document
Anticipate and explain the choices made in the Gluster patch series.Remove parts about a possible way of supporting userspace access asit has been implemented otherwise.
Signed-off-by: Santi Raffa <rsanti@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Fix 'htest' related targets dependencies
This fixes issue 634.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Design: Improvements of Node Security
This is a design doc addressing issue 377. Objective isto reduce the number of nodes that are able to establishssh and RPC connections to other nodes. Limiting thisset of nodes to the master candidates is desired to...
Make luxid inspect the job queue on startup
Since luxid handled the scheduling, make luxid also read the queueupon restart. In this way, jobs get scheduled in the same way, independentof luxid restarts.
Remove queue inspection from masterd
When masterd was responsible for queue handling, it would, onrestart, pick up all non-started jobs from the queue. Now thatluxid is responsible for handling the queue, make masterd notdo this any more.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Add a predicate to determine if a job has been started
Add a predicate jobs indicated that it has left the queue. Thiswill be needed, to allow restarts of luxid (which now handlesthe queue) independent of jobs (currently running in masterd).
Export getFStat from Utils
Use the jobFinalized predicate in JQScheduler
...to improve readability.
Provide a function to determine whether a job is finalized
While there is a function to calculate the job status, sometimesit is only relevant if the job is finalized. In this case, it ismore readable not having to know the internal order of JobStatus....
Don't assume we win the archive race
The job scheduler in luxid regularly watches for changesof the job files to determine progress of jobs. As thesefiles are updated atomically, reading them will alwayssucceed---until they're archived. While luxid is quite...
ganeti-mond: Add the "-b" option to the man page
This parameter was missing for this particular deamon and was requestedin issue #629.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
ganeti-mond: Add the "-b" option to specify the bind address
Support size suffixes in minmem/maxmem backed parameters
The backed parameters specifying the minimal/maximal memorycan also be passed as values with suffixes. Support parsingthese values.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Support fieldRead in partial params
While from parameters both full and partial versions are generated,with in the partial version all types mapped to Maybe, the fieldReadparameter of the field was not wrapped accordingly. So far, that didn'tmatter, as it was always Nothing in this case, but for supporting special...
Make disk size a special numerical field
For disk sizes, instead of plain numbers (naming the value in MiB),also accept expressions with units like 'GiB'.
Add a field-transformer for accepting parser
Add a transformer for numerical fields, to also acceptstrings instead of numbers if they can be parsed by thegiven parser.
Add a new unit parsing function taking all suffixes binary
In python, when parsing units (like disk sizes) we take allsuffixes (M, G, T) as 1024-based. To be backwards compatiblewhile moving to job management to luxid, in particular on RAPI,add such a parsing function in haskell as well....
Make JQScheduler handle failure on job starting
Given that luxid (at the moment) connects to masterd for startingjobs, it may be that this inter-process communication fails. Inthis case, just reschedule the jobs instead of killing the schedulerthread....
fix typo in log message
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Jose Lopes <jabolopes@google.com>
Differentiate watchers in luxid
luxid has two time-based watcher threads, one for theconfiguration, and one for the job queue. To improve readabilityof the debug output, make both watcher use a different debugmessage when the timer fires.
Make luxid use the JQScheduler
Make luxid use the job scheduler instead of immediatelystarting every received job.
Add a scheduler to keep track of the job queue
In order to allow informed decissions on when to start a job,it is necessary for luxid to keep track of the (active partof the) job queue. Add a scheduler, similar to the config reader,that does this, but also schedules jobs to be executed. At the...
Move FStat related function to Utils
In this way, the functions to to decide, based on fstat, whethera file needs to be reloaded can used by other parts as well,in particular to monitor progress in the job queue.
Rename enqueueJobs to startJobs
This reflects better what the method actually does. Later,we will add a job scheduler that will provide a proper enqueuemethod.
design: scheduling will be simple in the first iteration
In order to progress faster towards jobs as processes, the decision onthe final scheduling algorithm will be postponed. For the time being,the simple one will be used that starts jobs until a fixed maximal...
Add LUXID_ARGS to defaults/ganeti
luxid accepts arguments as well.
Parse spindles as integers
When parsing disk parameters, also convert spindles into integers.This is needed to obtain a syntactically correct request.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add default_iallocator_params cluster parameter
Add a cluster parameter to hold the iallocator parameters usedby the default instance allocator. Implement the option tomodify config.data, query config.data and upgrade man pages,tests and cfgupgrade tool. The new default_iallocator_params is...
Modify --mond to yes|no option
Modify --mond option used by hail, hbal and hinfo from nonargument to yes|no option.
Signed-off-by: Spyros Trigazis <strigazi@gmail.com>Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Remove parseUnit for spindles
In patch e33c9e6881e6, it was a bit overkill to useparseUnit for converting the 'spindles' value since it isonly an int without a unit. This patch fixes that.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Use query client for instance removal
This patch fixes another occurrence of triggering querieswith a non-query client.
Destroy Xen domain only if necessary
Check if a given Xen domain still exists after a 'xm shutdown' beforedestroying it. This check is necessary in order to cope with Xendomains created with other version of Ganeti, given that an instancecreated in or prior to Ganeti 2.10 will be cleaned up automatically...
Fix missing '@raise' in docstring of 'StopInstance'
Fix missing '@raise' in docstring of 'StopInstance'.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Activate QA for rapi queries via luxi
This patch enables QA testing for rapi queries for thenewly transformed queries from python to haskell(groups, instances, nodes, export, and networks). So far,the QA did not distinguish between resources that cannot be...
Set the received time stamp for new jobs
Since luxid now handles the job submission requests, it is alsoits responsibility to set the received time stamps. Do this.
Provide a function to set the received times tamp of a job
This is the pure function for changing the received time stamp;obtaining the actual time stamp has to be done in IO.
Document the jobqueue timestamp format
...and also provide a method to get the current time inthat format.
Design details of the implementation of new WConfD daemon
Describes steps needed to implement the new configuration and locksmanagement.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Fix integer parsing of spindles in gnt-instance add
'gnt-instance' did not parse the spindles value to integerbefore submitting the opcode.
Fix removal of duplicates
Commit ede6df3d02 introduced a bug in the node querieswhere disk templates where paired up wrongly to theirstorage unit keys due to removal of duplicates at thewrong place. This patch fixes it.
Signed-off-by: Helga Velroyen <helgav@google.com>...
Fix burnin error of instance name retrieval
Switching to Haskell queries revealed a bug in burninregarding the fetching of instance names from queries.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Fix retrieval of number of instances of a node
This patch fixes a FIXME to make the retrieval of thenumber of primary and secondary instances share morecommon code.
Use hypervisor / storage information only when requested
So far, the node queries ignored the list of fields andjust requested all available information from the backend.That means, for example if only hypervisor information isrequested, still the storage space calculation is...
Remove duplicate storage units in node query
This is a little performance tweak for the node queries.So far, the query code mapped the disk templates to storageunits. It could happen that two disk templates were mappedto the same storage unit and therefore the storage space...
build_chroot: for wheezy install base64-bytestring via cabal
The version packaged in debian wheezy is less than the minimallyrequired version 1.0.0.0.
Document the minimal version of base64-bytestring required
There exist several versions of base64-bytestring; the interface we'reusing is that of version 1.0.0.0 or higher.
Use the generic query field test for network QA
Instead of just using sporadic 'gnt-network list' commands,use the more exhaustive generic query test functions forlist and list-fields.
Update design doc wrt to queries from non-masters
This patch updates the design document of the querysplitting with respect to the proposal of making itpossible to run queries from other nodes than themaster node.
Mention removal of python queries in NEWS
This patch mentions the removal of the python queries inthe NEWS file, including the removal of the--enable-split-queries option at configure time.
Make luxid job submission be defined by replication
When receiving jobs to be submitted, make luxid replicate them to allmaster candidates and then return. The actual execution can be handledasynchronously.
Add function to enqueue jobs
Add a function that ensures that a given set of jobs gets executed atthe appropriate time. At the moment, this is still the simplemechanism of handing over everything to masterd; but even at thisstage, it has the benefit of allowing to remove code duplication in...
Add a function justBad to filter the Bad value of a list
In the same way as justOk allows to filter the Ok values,add justBad to filter the Bad values. While there, simplifythe definition of justOk.