remove duplicate code in Graph.hs
Also update the docstring of a function.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Log only partial response in Luxi when in debug mode
Currently, we log the entire response (at debug level) in the Luxireplies. This is not a good idea; the logging library operates onstrings, and as such it will use huge amounts of memory: without debug...
Jobs.hs: make L.Client the last argument for all functions
Having the Luxi client be the last argument of the functions allows foreasier use with `Exception.bracket L.getClient L.closeClient ...`.
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add shelltests for the mon-collector
This commit adds shelltests for the mon-collector binary and for the DRBDdata collector.
Also, it fixes a small bug in the DRBD parser found thanks to the tests.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Add the stand-alone data collector for DRBD
It is implemented as a single binary receiving, as its first parameter,the name of the actual data collector to be run.This way, its structure can be used for all the future data collectors.
Also, factored out of bdev.py into constants.py the location of the DRBD...
Network and address pool (again)
Implementation of the network objects and address pool. Functionalityas in the reverted commit b9a616e132af9bba718d2da8c94eeec7af886814, butnow using only the vector library and not the bit-vector library.Tested with vector library version 0.9 and 0.10., which are...
htools/Ganeti/Errors: Add ECodeTempNoRes
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: Add missing parameter to OpInstanceMultiAlloc
Somehow this went missing in commit 1f1188c.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
cmdlib: Opportunistic locking on instance creation
Adds a new parameter to “OpInstanceCreate” and “OpInstanceMultiAlloc” touse opportunistic locks.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix typo in comment
There was a typo in the docstring of the parseMessage function of theconfd server.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Revert "Network and address pool"
This reverts commit b9a616e132af9bba718d2da8c94eeec7af886814, whichdepends on the "bitvec" library. We need to investigate a bit further thatdependency, as it in turns depends on "vector" 0.9.1 or below, since0.10.* removed support for Data.Vector.Unboxed.Safe which it uses....
Network and address pool
Implementation of the network and address pool class inHaskell. Not complete yet. Includes unit tests that coverall functionality that is so far implemented.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Hbal.hs: move job execution functions to Jobs.hs
Ganeti.Jobs now holds functions that can be used to submit and monitor thestatus of jobs. In particular, execJobsWait and waitForJobs are factoredout of Hbal.hs.
Signed-off-by: Dato Simó <dato@google.com>...
Hbal.hs: use Result (), not Bool, as return value for exec*
Previously, functions in Hbal.hs related to execution of jobsets werereturning only IO Bool, and printing any errors they found directly tostderr on their own.
I'm going to be moving some of these functions to a library module in...
Hbal.hs: rename functions that deal with early termination
In Hbal.hs, it's easy to get lost in the flow of exec* functionsbecause their names are similar and don't convey their differentpurposes (e.g. runJobSet, execJobSet, execWrapper).
This patch renames 'runJobSet' to 'execWithCancel', and 'execWrapper'...
Hbal.hs: fix typo in signal handler names (hangle -> handle)
'hangleSigInt' and 'hangleSigTerm' are renamed to 'handleSigInt' and'handleSigTerm', respectively.
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Jobs.hs: move OpStatus and JobStatus ADTs to Types.hs
This leaves Ganeti/Jobs.hs and Test/Ganeti/Jobs.hs empty, but they're thetarget of a future move of some functions, so we leave them around, anddon't delete them, to avoid unnecessary delete/create diffs....
Add Ganeti.HTools.Graph
This module implements some algorithms on Data.Graph data structures.At the moment its main functionality is an LF-color implementation(greedy coloring in descending order of degree). There are also a fewextra functions to calculate the degree order, and convert the node to...
Add Dsatur implementation
Implement the Dsatur algorithm for Graph coloring. This also abstractsthe neighColors function into two subfunctions that this algorithm canreuse.
Ganeti/HTools/Graph Add isColorable
Check whether coloring on a given graph makes sense. This is the caseonly if there are no loops and the graph is undirected.
HTools/Node: add mkNodeGraph function
This function helps treating node node problems as graph problems.It can transform a list of nodes plus a list of instances into a graphwhich uses the nodes as vertices, and instances as edges connecting them(as long as they have both a primary and a secondary node)...
Fix Dsatur and add Dcolor
Our Dsatur implementation was incorrect: while the paper defined thedegree of saturation of a vertex as the number of different colors it isadjacent to, we were using the number of colors, without consideringuniqueness. This effectively implemented a different algorithm, which is...
Add "proper coloring" unittest check
We have to check that for each edge its vertices have different colors.
This is very easy to do with a vertex-to-color map, but not so easy witha color-to-vertex one. Since all our coloring algorithms created avertex-to-color map behind the scenes and then converted it, we flip...
Small improvement to rpc-test
This will show smaller/better error messages: full node dumps are nolonger included.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Switch luxi submit job calls to use MetaOpCode
This patch changes the luxi submit job calls to use wrapped opcodes,and therefore it changes Hbal to submit actual meta opcodes. Fornicety, hbal also submits a comment now, showing who generated thejob.
Signed-off-by: Iustin Pop <iustin@google.com>...
hbal: restore nice job ID display
This fixes:
Got job IDs JobId {fromJobId = 1052613}
And restores to:
Got job IDs 1052624
Other improvements could be done here, of course.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add a negative type
This mirrors the positive one, and will be needed for relative jobIDs.
Add types and parameters for common opcode implementation
This will go into a separate type; this patch adds the neededunderlying types and parameters.
Add a type alias for simpler THH signatures
This is reused in more than just a few places, so adding it makes thesignatures much nicer.
Change opcode/luxi showJSON generation in THH
Currently, the opcode and luxi "showJSON" functions generate directlya JSValue; in contrast, the object (single-constructor) types generatea 'toDict' function, and then `showJSON = makeObj . toDict`. This is...
Remove unused THH function 'saveLuxiField'
This was deprecated via commit 88609f00, “Switch Luxi TH code fromsimple to custom fields”.
Add CommonOpParams and MetaOpCode types
This patch adds the "meta" opcode type and the common opparams. Compatibility tests with Python are changed to pass Metaopcodes.
Fix long lines in Ganeti.HTools.Types and add Make check
Since hlint doesn't check style issues but rather code only issues,let's add a simple Makefile check for too-long lines, and fix thesingle bad case we have currently.
Rework custom fields handling
This patch changes a bit the handling of custom fields. Since ingeneral we use custom fields to aggregate multiple entries in the JSONobject into a safer data-type, we should also have a way to declarewhich extra entries this field covers (so that in the future we can...
Add a function for all fields of a given OP_ID
This patch changes THH to export a new function which defines allfields of a given OP_ID. Not very clean, since for an invalid OP_ID wereturn empty list, but since it will only be used in tests it shouldbe good enough....
Rework/enhance original htools opcodes
The original htools opcodes were minimalistic and not 1:1 equivalentwith the Python ones. Let's add all missing fields and, since wechanged the order, switch to more readable record syntax for buildingthe opcodes....
Add a few missing fields from opcodes
Due to manual conversion, a few fields were missing from theconversion, but as they were optional our type equivalence checkingdidn't detect this.
Fix Haskell OpNetworkAdd
Commit 213076f (“Fix locking in networks”) changed Python OpNetworkAddwithout corresponding Haskell definition changes.
Add type for finalised job statuses
For now, we don't need a pending job status type as well, so we'lldelay adding that until later.
Add a 'real' type for JobIds
Currently, the job ID is a simple type alias. This is suboptimal, asit means we can't use a custom JSON (or Arbitrary) instance for it.
The patch changes it into a newtype, and then a) simplifies somedeserialisation code and b) changes some more fields to this new type...
Fix variable names in Loader.assignIndices
This function can be applied to any type of class "Element" but itsinternal variables used to refer to a list of tuples containing theseelements and their names as "nodes". Moreover it built a list of name toindices tuples referring to it as "na" (which is not very clear)....
Add two support functions for building JSON objects
The new support functions in module Ganeti.JSON will be used by the nextcommits to build JSON objects with optional (Maybe) fields, excludingthem when they contain Nothing.
Add JSON serialization capabilities for DRBD status
The serialized JSON is not a 1:1 dump of the data structurespopulated by the parser. This is done intentionally, with theaim of producing a more stable and more meaningful output tobe used by the (future) monitoring agent and stand-alone data...
Add unit tests for the JSON serialization of DRBD status
The serialization itself is done by Text.JSON, so the tests deal with checkingthat Text.JSON objects are created correctly from the DRBD parser datastructures.
Change -G option description
The new one is more consistent with the rest of the nodegroupterminology and more generic for use in different htools.
Make Query operators enforce strictness
Currently, the query operators (binop, etc.) create thunks, instead offorcing the evaluation of the simple boolean results. This results inhigher than needed memory use.
Optimise recursive Query filters
Currently, the And and Or filters use very nice code, e.g. in case ofOrFilter:
any id <$> mapM evaluateFilter flts
However, looking at the memory profiles shows that application ofany/id to monadic values via '<$>' does not work nicely, losing the...
Fix htools/rpc-test after recent changes
Since this program is not built normally, a few recent commits brokeit:
- 707cd3d (“Use exitErr instead of explicit error message and exitWith”) introduced wrong syntax (missing '$')- 29a3053 (“Remove unsafePerformIO usage”) changed the signature of...
Add two last missing opcodes
OpInstanceQuery was missing accidentally, whereas OpRestrictedCommandwas just recently added without Haskell definitions.
The patch also slightly improves the OpNodeQuery arbitrary generation.
Abstract query common opcode definitions
It would be even better if the opcodes would actually have all thesame definitions, until then we have two sets of definitions.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Remove remaining in-line parameters
… except one, and replace them with separately-defined ones inOpParams.
Replace hand-coded 'live' field of OpInstanceMigrate
This is the last inline-written field; I've done it separately fromthe last patch due to changes being needed in HTools (field changedtype).
Remove read instances from our Haskell code
It turns out that optimising 'read' derived instances (via -O) forcomplex data types (like OpCode, or the various objects) can be slowto very slow. Disabling such instances results in (time make$all_our_haskell_binaries) large compile-time savings and also smaller...
Complete the instance OpCodes and parameters
Only the original instance opcodes (used in htools) are leftnon-converted to only parameter style; they'll be cleaned up later,once the htools codebase itself migrates to safer types.
Add Group, OS and Backup opcodes
This also corrects a docstring in OpBackupExport on the Python side.
Complete the Tag opcodes and fix OpTestDelay missing param
The regexp in OpTagsSearch is loaded as is, without testing forvalidity; the rest of the patch is trivial.
Complete the Test opcodes
This adds the OpTestAllocator, OpTestJqueue and OpTestDummyopcodes. The OpTestAllocator seems to need some cleanup (on the Pythonside), for now we implement it as is. As for the other two, while notused in production, we should have full coverage for them as well....
Move NICMode from Objects.hs to Types.hs
Also add some unittests for this type.
Add types, parameters and the opcodes for networks
This completes the last missing opcode group. The only difficulty waswith the ip addresses, where we used simple strings to represent themand (for IPv4) a few helpers to generate arbitrary instances;otherwise, the patch is trivial....
Add types and parameters for OpInstanceCreate
This is a "big" opcode, so sending it separately.
A few types needed changing, and a few parameters were renamed to makeit more clear which are cluster-level and which are instance-levelparameters.
Move FileDriver from Objects to Types
Another base type that we need in opcodes as well.
Small improvements to the DRBD parser and tests
Implement the correct handling of numbers without commas
commaInt now recognizes only the first 3 digits for numbers without commas.
It was erroneously recognizing numbers of any size before the first comma.
A few more hlint fixes
I'm ignoring the use of Control.Arrow.*** since it's a non-obviousfunction, and we're not widely using Arrows in the code base, so Ithink switching to it would make the code less readable to mostpeople.
Bulk add of remaining cluster/node opcodes
These are tested for equivalence with the Python code using existingtests. These tests are not perfect - for many "TDict" we simply sendempty dicts, as we don't have a way to generate proper data, but thetests did catch many trivial errors during writing of this patch...
Convert a few existing opcode parameters to safer types
This does a partial conversion of existing opcodes to the newnon-empty string type, which is needed so that we don't add even moreopcodes without this safety net (and to have consistent "target_node",...
Add many more opcode parameters
This is a bulk add of all node parameters needed for node and clusteropcodes. The parameters are defined with a few helper functions inthis module, and, opposite from opcodes.py, there won't be any (new)opcode attributes declared in-line (without a separate field...
Add two more basic types
These are needed for completing the node opcode parameters.
Move StorageType from RPC.hs to Types.hs
This fixes a FIXME actually recommending this move :)
Split 'Query.Language.ItemType' in two sub-types
The QR_VIA_OP/QR_VIA_LUXI types in Python are using yet anothervalidation mode: QR_VIA_OP is the base type, and QR_VIA_LUXI extendsit (when doing luxi queries). But on the wire they have the samerepresentation....
Move the 'Hypervisor' type from Objects to Types
This is a very basic type and 'Objects' is a heavy-weight module. Bymoving it to 'types' we simplify (in the future) the import chains.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Adeodato Simo <dato@google.com>
Generalise the JSON "Container" type
Currently, we have some types that we kept as dictionaries in theobjects representation (due to inconsistent behaviour, or othertechnical reasons). This should be improved in the future, but in themeantime we can improve the Container type by allowing its keys to be...
Add more basic Ganeti types
These are needed both in opcodes and (potentially) outside, so we addthem in the main Types module.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add more basic validation types
This mirrors the ht.py types PositiveInt, NonNegative, etc., exceptthat they work at a more generic level (any numeric type, respectivelyany non-empty list).
Add parser for DRBD /proc file
Properly log errors when setting up daemon FDs
While writing the pipe-based reporting and trying various ways tobreak the startup, I fought for a while trying to understand why errorreporting was different when running the daemon as a user (with norights). It turns out that setupDaemonFDs wants to open the log file...
Switch opcode data type from normal to record constructors
Currently, the OpCode definitions are using normal constructors:
data OpCode = OpTestDelay Double Bool [String] | OpInstanceFailover String Bool (Maybe String) …
While this works for a few opcodes, it becomes unwieldy when dealing...
Fix docstrings for the Filter type
While looking at the opcode docs and clicking accidentally on thefilter type, I saw that the haddock formatting is broken due tonon-escaped use of special chars. Let's convert the ascii-likeformatting to haddock, and have nicer apidoc....
Split OpCode.hs and add module for opcode parameters
Due to TemplateHaskell stage restrictions, we can't define parametersin the same module as we're using them for TH, so we have to defineall module parameters in a separate module.
This patch therefore splits OpCodes.hs in two, adding that module and...
Create a new Ganeti/Types.hs module
There are already three cases where we copied type definitions betweenthe htools-specific types into the main ganeti code. Let's stop doingthis ☺ and create a common types module that holds these.
Note that there already exists BasicTypes.hs, but that refers to very...
Cleanup THH function use from built module namespace
Currently, THH.hs "injects" into the built code names of libraryfunctions like Text.JSON.makeObj, Ganeti.JSON.fromObj, etc. builtdirectly from strings, via (e.g.)
varE (mkName "makeObj")
This means that the "makeObj" name must exist in the target module,...
Expose changing job priority via LUXI
A new LUXI request is added, in both Python and Haskell.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix two hlint warnings
Sorry, I broke lint again :), by introducing two sub-standard changes.
Additionally, this silences an older existing warning that onlytriggers with some versions of hlint (e.g. 1.8.28 which is present inWheezy).
Merge branch 'devel-2.6' into master
Conflicts (both trivial): htools/Ganeti/Daemon.hs (_writePidFile rename)...
Merge branch 'stable-2.6' into devel-2.6
Fix PID file writing in Haskell daemons
Currently, the code uses createFile, which has the effect of alwaystruncating the file. This is bad, as the content of the PID file iswiped even when we wouldn't be able to lock it!
We switch to openFd (createFile is just a wrapper over that), and we...
Remove unsafePerformIO usage
We need to change a few things, most importantly CLI options defaults,but otherwise we already used the path to files from functions whichwere already in the I/O monad, so we don't have to change much of thecode flow.
Additionally, Path.hs now has an explicit export list, to stop...
Change keyToFilename/readSSconfFile interaction
Currently, keyToFilename uses itself the default config path. In thelight of making that an function in the IO monad, let's remove the"default" path functionality from this function and make it alwaysrequire the config path; its caller, readSSConfFile, can then do the...
Change type of program options to 'IO [Options]'
Some options have defaults that depend on the environment, and wecould handle these in two ways:
- use a place-holder value (e.g. data X a = Default | Custom a) that is later read from the environment- move the options list to IO monad, where it can read the...
Two small logging improvements
The "starting" message is changed to match the Python one, and in casethe preparation fails, we also log the error (beside printing it onstderr or writing it to the error reporting pipe), as at this timelogging is usually set up....
Annotate confd startup/luxi binding error
The luxi binding is now annotated for better readability.
Switch writing of PID file to new reporting style
Currently, the writing of the PID file uses a "standard" errorreporting: catch exception, transform it into a 'Result' type, leavehandling of that to the caller. However, for daemon startup, weactually want exceptions to be propagated up until the handler which...
Move and rename BasicTypes.annotateIOError
It turns out that annotateIOError already exists in the standardlibrary, with a different purpose (this made me waste 10 minutestrying to understand why the types were wrong…). Let's move this outof BasicTypes, since it's used only in Daemon.hs, and also rename it....
Switch daemon startup to pipe-based error reporting
This changes the daemon reporting error to the same mechanism as inPython: use a pipe which is written to from the forked children.
Move preparation steps of confd in prepMain
This does the address parsing earlier, before we fork, forbetter/faster error reporting.
Split queryd run into prepare and exec
This will help with the general daemon split of prepare/run, and flagerrors earlier in the startup.
Convert tag objects to a safer type
Currently, we keep information about the "target" of a tag operationin a data type similar to (TagKind, Maybe String). This is unsafe, asnothing (at the type level) prevents us from accidentally having(TagCluster, Just "instance1.example.com"), or (TagInstance, Nothing)....
Switch Luxi TH code from simple to custom fields
This is needed so that we have more flexibility in generating Luxiserialisation code (deserialisation is still custom). Also, onlyexceptions are now using the 'simple' field types, so we might be ablelater to convert and remove that TH code as well....
Make THH:genSaveOpCode a bit more general
This can be improved, by taking all hardcoded names as parameters, toserve as a more-general "build save clause for a multi-constructordata type". I'm not renaming the function as well, since I don't knowexactly how much we can abstract later....
Rename Confd.hs to Confd/Types.hs
This should be the last module rename, promise!
We rename this to conform to the other hierarchies (e.g. Query), andto not have both Confd.hs and Confd/*.hs.