Improve TemplateHaskell code to support empty objects
Currently, an empty objects will generate warnings as the arguments ofvarious functions are unused. By adding conditional code for this, wecan support generation of empty objects, e.g. like needed in Rpc code....
Switch the curl bindings from optional to required
Currently, we support curl being optional via some sporting exercises:ifdefs in the code, data types that represent 'Curl is disabled'state, etc. However, with the future work on RPC, we would have toeven make the dependencies list conditional on it, etc. This is too...
Reduce duplication of curl options computation
Some curl option are request-specific, but not node specific: e.g. rpctimeout, etc. The patch changes the HttpClientRequest type so that wecan pre-seed such options, instead of rebuilding the list in eachindividual request execution....
Simplify RPC error cases
This patch removes the node from the RPC error constructursCurlLayerError and OfflineNodeError. The rationale is that we anywayreturn tuples (node, result), and removing this duplication allowssimplified signatures/calls in the execution of RPC calls....
Add two utility functions for handling Either lists
These two functions permit operating in bulk on only the Left or Rightvalues in the original list, then reassembling the list back in theoriginal order.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Add a Ganeti-specific implementation of Curl Multi
As we want to be able to run queries against multiple nodes inparallel, and furthermore in parallel with other work, we need toimplement the Curl Multi interface (see libcurl-multi(3)).
This patch adds a Ganeti-specific such implementation, to be used...
Switch the RPC module over to the multi interface
This replaces the very-basic parMap of IO actions (fully serialised,as parMap won't work here), to the multi interface.
This makes a simple "time gnt-node list" on a 6-node cluster go from3.2s to ~0.9s, and allows even better parallelisation - before,...
Status change reason support for Reboot
Add support to the Reboot command for specifying the reason for the laststatus change.
Some features are implemented as functions, even if used only once, becausethey will be used by the future patches introducing reason support for all...
Infrastructure for specifying instance status change reason
This patch introduces some infrastructural modifications that will be used bythe following commits to implement the support for specifying the reason forthe last status change of an instance....
Unit tests for Query/Network.hs
This patch adds a couple of unit tests for Query/Network.hs.Note that they'll need to be adapted, once issue 362 is addressed.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Make Confd client usable for testing
Allow the Confd client to be able to connect to an arbitrary serverinstead of just the real one running on a cluster.
This is meant to be used for testing.
If a data collector using the Confd client is tested with shelltest, it will...
Add Haskell parser for "xm uptime"
In order to fetch precise information about the uptime of the VMsrunning in Xen, we need to analyze the output of the "xm uptime" command.
This commit adds the parser to do that, and its tests.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Add Haskell parser for "xm list --long"
In order to fetch precise information about the status of the VMs running inXen, we need to analyze the output of the "xm list --long" command.
Add request type to Confd server for getting instance list
Add to Confd server a new request type (and its implementation) to ask forthe list of instances in a node.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge branch 'devel-2.7'
Add a function to change an OpCode's priority
This simply updates the metaopcode submit priority.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add functions to parse CLI-level format of priorities
The current serialisation format for submit priorities isinteger-based, same as the opcode json serialisation. But for CLIlevel, we need to support a string-based format, so we add functionsto parse and format this representation....
Add CLI-level option to override the priority
This just defined the new priority, with the same name as the Python one.
Enable use of the priority option in hbal
This patch adds the option to hbal, and uses it to tweak the submittedjobs. There are also two small shelltests for testing the parsing.
Make hbal opcode annotation more generic
Currently, hbal code always uses annotateOpCode function, which meanswe would have to pass the options data to all function in the callchain if we wanted to make this more flexible.
By abstracting the type of the annotator and passing it as an argument...
Removes check for conflicts from NetworkDisconnect
This removes the check for conflicts from the Haskellversion of the OpCode NetworkDisconnect. This alignesthe Haskell code with the patch"Force conflicts check in LUNetworkDisconnect" (whichis currently under review). I will submit these patches...
Remove network_type slot (Issue 363)
This slot was not used by Ganeti so the same info can beprovided via tags. In order not to break configuration datawe add a FromDict() method in Network config object thatremoves the deprecated network_type (if found) and then invoke...
Remove family and size from network objects
This info is not used by Ganeti and therefore is removed.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Guido Trotter <ultrotter@google.com>
Make ParticalNic's network field of type String
This was applied to "master" along with extra changes affecting themaster branch only. Cherry-picking just the Objects.hs change.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Remove use of 'head' and add hlint warning for it
Since 'head' is unsafe to use in most cases, this patch removes itsuse from most of the code, adds a lint warning for it (and for tail aswell), and adds override annotations in the few cases where it'sactually OK to use it (mainly when using head over the result of...
Harep.hs: fix a couple typos in comments and docstrings
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix improperly formatted docstring
Change the docstring of chompPrefix to prevent the error"doc comment parse failed" that was raised by some version ofhaddock while generating the documentation for this function.
Making ParticalNic's network field of type String
This is yet another fix for type confusion between pythonand haskell. ;) The network field of PartialNic should bea string and not of type Network. This makes it necessaryto add a helper function to look up a network by name...
Fix typo in a comment
Fix Haskell log file naming after virtual cluster changes
Commit 3329f4de changed the Haskell log file from constants tofunctions, but introduced a bug: it uses now the daemon name insteadof the correct log file, which means "ganeti-confd.log" instead of...
harep: create repair jobs
Implement 'doRepair' to create a repair job from a list of opcodes ifthe instance's policy allows it (otherwise set an ENOPERM result label),and the instance was previously healthy (i.e. not in ArFailed orArPendingRepair)....
harep: do not wait for repair job completion to set tags
Because of instance locks, after submitting a repair job we weren't able toset the "pending" tag until at least the first opcode of the job finished.Introduce a small delay in the repair job so as to allow the subsequent...
harep: finish execution with a summary of instance states
The harep tool prints messages for every action that it performs (or thatit doesn't perform). In case nothing is to be done at all, always printsome statistics of the current state of the cluster....
harep: initial parsing of tags
Parse auto-repair tags to set each instance in one of ArHealthy, ArFailed,or ArPendingRepair. The implementation tries to be well behaved when oldtags have been left behind, which future patches will still try not to do....
harep: check for completed jobs at the start of the program
As a first step before detecting any brokeness with instances, see if anyof our previous repairs have completed, and move instances to ArFailed orArHealthy accordingly. Do nothing if there are still running jobs for the...
harep: pure function to detect brokeness with instances
Add a 'detectBroken' function that determines whether an instance is in anunhealthy state, and what's needed to repair it. The repair is specified asan AutoRepairType constant, and a list of opcodes. The opcodes will only be...
Program/Harep.hs: add skeleton for the new auto-repair tool
harep(1) detects certain kind of problems with instances and applies theallowed set of solutions. See doc/design-autorepair.rst.
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix typo ("Abrreviation" -> "Abbreviation" in Common.hs)
Jobs.hs: add a submitJobs function
This new 'submitJobs' function doesn't wait for job completion, it justreturns the job IDs immediately.
Jobs.hs: add an execJobsWaitOk function
This new 'execJobsWaitOk' returns "Bad" not only if an error occurs, but ifany of the submitted jobs doesn't succeed.
Jobs.hs: start with a shorter delay in waitForJobs
Instead of waiting 15 seconds every iteration of the waiting loop, startwith 0.5 seconds and increment exponentially until a certain maximum isreached. Keep the maximum at 15 seconds.
Signed-off-by: Dato Simó <dato@google.com>...
HTools/Types.hs: minor adjustments to auto-repair types
In particular:
- make ArHealthy take an optional AutoRepairData; this allows to represent the situation where a repair completed successfully, and hence there's an associated tag we might want to know about....
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
CLI.hs: fix double spaces in option help strings
Some help strings with continuation backslashes ('\') were providing aspace both before and after the backslash, resulting in double spaces inhelp output. Provide it only after the backslash, which fixes the issue and...
Add fields 'inst_cnt' and 'inst_list' to network queries
This adds the fields 'inst_cnt' and 'inst_list' to theHaskell implementation of the network queries.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Moving network query helper functions to Network.hs
So far, a couple of helper function for the networkqueries resided in Config.hs. I figured it makes moresense to move them to Query/Network.hs, since they arevery tailored to that purpose.
Signed-off-by: Helga Velroyen <helgav@google.com>...
More fields for network queries
This adds more fields to the network queries:- group_cnt, free_count, reserved_count, and map
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
First part of Network Queries in Haskell
This is the beginning of the implementation of networkqueries. This includes establishing all infrastructureto run the network queries and implement querying ofsome simpler fields and the node group listing.
Convert Maybe results to RSUnavail
When displaying query results of type Maybe, one could use thefunction rsMaybe. Unfortunately, this function maps 'Nothing'values to RSNoData which gets displayed as '?' in the list ofquery results. These semantics do not fit if the result is of...
Extend config by networks and networks by UUIDs
For network queries to work, we need to extend the generalconfig type to include all available networks. Additionally,we add UUIDs to the network type itself.
Typo in comment of network type
Renames and cleanup of variable names in confd
The current names are quite confusing; this patch cleans up theconfusion by making sure we use different terms for the two threads,etc.
No actual code changes besides the renames.
Signed-off-by: Iustin Pop <iustin@google.com>...
Add a function to get the current time in microseconds
In some cases we need higher resolution that seconds; I've settled onmicroseconds as that is what 'threadDelay' wants, for exactly, so it'seasier if we keep the same units.
Increase Haskell log file resolution
Currently the Haskell logging goes with the default time formatting,which lacks sub-second resolution. This is not good, as it's toocoarse.
The patch adds picoseconds to the log; they will contain usually toomany zeroes, but the other option is to use a log format that (for...
First part of confd timer changes
This patch changes the resolution of the timers: the watcher timergoes from 60s to 17s, and the polling-mode timer goes from 2 secondsto 250ms. The code changes a bit more due to the changes in the unitsof the various constants....
Change the polling thread sleep behaviour
Currently, the polling thread sleeps at the end of its body; theoriginal reason was that when the thread initially starts, we want toload the config immediately (in the old-style config load model).
However, now we explicitly load the config, so we can make this thread...
Start confd in notify mode if we can
This patch changes the default model—where confd is always started inpolling mode—to a model where if possible we enable inotify modebefore starting any of the background threads.
There are some tricky details here: if we enable inotify, we should...
Explicitly load the config at confd startup
Currently, confd main function doesn't load the config, but simplyforks the individual timer threads, which will—as a side effect-alsoload the configuration as soon as they start.
However, this makes codes to the startup more complex, so let's...
Text wrap in Haskell code
This adds two functions in Utils.hs: 'wrap' and 'trim' along with somesimple tests. 'wrap' is used in Common.hs for wrapping commanddescriptions to a fixed line length, e.g. useful for the "--help" option.
Signed-off-by: Yiannis Tsiouris <gtsiour@softlab.ntua.gr>...
Integrate DRBD with the report generation infrastructure
With this commit, the DRBD data collector does not only print the datait extracts, but includes them in the proper JSON structure common toall the data collectors, as prescribed by the design document....
Add reporting infrastructure for data collectors
This commit adds the part of the JSON report generation code that will becommon to all the data collectors, according to the format specified in thedesign document.
Move getCurrentTime from Confd.Utils to Utils
The getCurrentTime function is of wide interest, so its moved to a moreappropriate position.
Make niceSort a wrapper over niceSortKey id
Currently, niceSort behaves differently than niceSortKey in caseswhere the parsed groups are equal but the original not, as it usesboth members of the tuple as opposed to just the first one. BothniceSortKey and the Python code use only the first one, so in corner...
Replace nodegroups' PartialNic by PartialNicParams
This fixes a bug that corrups the config when one adds anetwork to a node group. I wil soon extend QA by some commandsthat would have detected this.
Loader.hs: ignore expired ArSuspended policies
At the moment, because 'mergeData' is pure, it may set instance auto-repairpolicies that are of the form `ArSuspended $ Until timestamp_in_the_past`.If later on the auto-repair tool notices this, it has lost access to what...
Loader.hs: rewrite extractExTags to use chompPrefix
Loader.hs: set instance auto-repair policy in mergeData
'getArPolicy' and 'setArPolicy' follow the precedence rules introduced inb1eb71c: within an object, the most restrictive tag wins; across object,the nearest tag wins.
Instance.hs: add an 'arPolicy' field for auto-repair policy
Utils.hs: add a clockTimeToString function
This function allows to easily convert a ClockTime object to a stringrepresentation of its timestamp (seconds-only).
Utils.hs: function to chomp prefix + separator from a string
Some auto-repair tags are composed of a prefix and then an optionalargument; the new "chompPrefix" function in Utils.hs allows to strip aprefix, allowing the last character (the separator) to be absent if there...
Add initial constants and Haskell ADTs for auto repair
In this commit, the AutoRepairType and AutoRepairResult types are defined,with the possible values specified in doc/design-autorepair.rst.
HTools/Types.hs: more auto-repair types
AutoRepairPolicy, AutoRepairStatus, and other auxiliary types are added.These are used only internally by the auto-repair tool, and parsed from thevarious object tags as defined in the design doc.
Fix typo in the DRBD parser types file
A missing letter has been added to a docstring.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Support instance-minor pairing in the DRBD collector
This commits enables the DRBD data collector to use the Confd client togather information about the pairing between DRBD minors and instances.
For testing purposes, the DRBD data collector now requires either zero...
Support integrating instance information in the DRBD parser
This commit modifies the DRBD parser and its data structures to includeinformation about the instance a DRBD minor belongs to.
Test files have been updated as well, to support the new field in the data...
Fix a bug in the generation of completion information
Completion information for multi-command binaries were not generated inthe correct way, so the build-bash-complation script failed when such abinary was actually found.
This commit fixes this issue, and a related docstring typo as well....
Add "use_locking" parameter to network query opcode
This was extracted from a patch by Dimitris Aragiorgis with the subject“Add locking to _NetworkQuery”. The rest of the patch did no longerapply and will require more work.
Improve the reception of replies in the confd client
If a received reply is valid and there is still no valid reply,keep it, no matter how new it is.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix a bad data type in Hcheck.hs
While trying to understand why some code was not being tested, Irealised that we have a bad data type in Hcheck.hs.
We have "data Level = GroupLvl | ClusterLvl", but then we need to passthe group name/index as well, so we have functions that look like the...
Move src/Ganeti/HTools/Program.hs to Program/Main.hs
This removes one more tab conflict; this is the last module in ourcode where we have both x.hs and x/.
Furthermore, we collapse all actual code into the new Main.hs module,leaving the htools.hs basically empty (will allow better testing in...
Rename htools/ to src/
Per offline discussions, this is the first patch of therenames. Tested with "make distcheck", seems to work fine.
The only change outside of the renaming is a bit of simplification inthe .gitignore rules; otherwise, simply s/htools/src/....