Add Ganeti.HTools.Graph
This module implements some algorithms on Data.Graph data structures.At the moment its main functionality is an LF-color implementation(greedy coloring in descending order of degree). There are also a fewextra functions to calculate the degree order, and convert the node to...
Add Dsatur implementation
Implement the Dsatur algorithm for Graph coloring. This also abstractsthe neighColors function into two subfunctions that this algorithm canreuse.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Ganeti/HTools/Graph Add isColorable
Check whether coloring on a given graph makes sense. This is the caseonly if there are no loops and the graph is undirected.
HTools/Node: add mkNodeGraph function
This function helps treating node node problems as graph problems.It can transform a list of nodes plus a list of instances into a graphwhich uses the nodes as vertices, and instances as edges connecting them(as long as they have both a primary and a secondary node)...
Small improvement to rpc-test
This will show smaller/better error messages: full node dumps are nolonger included.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Switch luxi submit job calls to use MetaOpCode
This patch changes the luxi submit job calls to use wrapped opcodes,and therefore it changes Hbal to submit actual meta opcodes. Fornicety, hbal also submits a comment now, showing who generated thejob.
Signed-off-by: Iustin Pop <iustin@google.com>...
hbal: restore nice job ID display
This fixes:
Got job IDs JobId {fromJobId = 1052613}
And restores to:
Got job IDs 1052624
Other improvements could be done here, of course.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add a negative type
This mirrors the positive one, and will be needed for relative jobIDs.
Add types and parameters for common opcode implementation
This will go into a separate type; this patch adds the neededunderlying types and parameters.
Add a type alias for simpler THH signatures
This is reused in more than just a few places, so adding it makes thesignatures much nicer.
Change opcode/luxi showJSON generation in THH
Currently, the opcode and luxi "showJSON" functions generate directlya JSValue; in contrast, the object (single-constructor) types generatea 'toDict' function, and then `showJSON = makeObj . toDict`. This is...
Remove unused THH function 'saveLuxiField'
This was deprecated via commit 88609f00, “Switch Luxi TH code fromsimple to custom fields”.
Add CommonOpParams and MetaOpCode types
This patch adds the "meta" opcode type and the common opparams. Compatibility tests with Python are changed to pass Metaopcodes.
Fix long lines in Ganeti.HTools.Types and add Make check
Since hlint doesn't check style issues but rather code only issues,let's add a simple Makefile check for too-long lines, and fix thesingle bad case we have currently.
Rework custom fields handling
This patch changes a bit the handling of custom fields. Since ingeneral we use custom fields to aggregate multiple entries in the JSONobject into a safer data-type, we should also have a way to declarewhich extra entries this field covers (so that in the future we can...
Add a function for all fields of a given OP_ID
This patch changes THH to export a new function which defines allfields of a given OP_ID. Not very clean, since for an invalid OP_ID wereturn empty list, but since it will only be used in tests it shouldbe good enough....
Rework/enhance original htools opcodes
The original htools opcodes were minimalistic and not 1:1 equivalentwith the Python ones. Let's add all missing fields and, since wechanged the order, switch to more readable record syntax for buildingthe opcodes....
Add a few missing fields from opcodes
Due to manual conversion, a few fields were missing from theconversion, but as they were optional our type equivalence checkingdidn't detect this.
Fix Haskell OpNetworkAdd
Commit 213076f (“Fix locking in networks”) changed Python OpNetworkAddwithout corresponding Haskell definition changes.
Add type for finalised job statuses
For now, we don't need a pending job status type as well, so we'lldelay adding that until later.
Add a 'real' type for JobIds
Currently, the job ID is a simple type alias. This is suboptimal, asit means we can't use a custom JSON (or Arbitrary) instance for it.
The patch changes it into a newtype, and then a) simplifies somedeserialisation code and b) changes some more fields to this new type...
Fix variable names in Loader.assignIndices
This function can be applied to any type of class "Element" but itsinternal variables used to refer to a list of tuples containing theseelements and their names as "nodes". Moreover it built a list of name toindices tuples referring to it as "na" (which is not very clear)....
Add two support functions for building JSON objects
The new support functions in module Ganeti.JSON will be used by the nextcommits to build JSON objects with optional (Maybe) fields, excludingthem when they contain Nothing.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Add JSON serialization capabilities for DRBD status
The serialized JSON is not a 1:1 dump of the data structurespopulated by the parser. This is done intentionally, with theaim of producing a more stable and more meaningful output tobe used by the (future) monitoring agent and stand-alone data...
Add unit tests for the JSON serialization of DRBD status
The serialization itself is done by Text.JSON, so the tests deal with checkingthat Text.JSON objects are created correctly from the DRBD parser datastructures.
Change -G option description
The new one is more consistent with the rest of the nodegroupterminology and more generic for use in different htools.
Make Query operators enforce strictness
Currently, the query operators (binop, etc.) create thunks, instead offorcing the evaluation of the simple boolean results. This results inhigher than needed memory use.
Optimise recursive Query filters
Currently, the And and Or filters use very nice code, e.g. in case ofOrFilter:
any id <$> mapM evaluateFilter flts
However, looking at the memory profiles shows that application ofany/id to monadic values via '<$>' does not work nicely, losing the...
Fix htools/rpc-test after recent changes
Since this program is not built normally, a few recent commits brokeit:
- 707cd3d (“Use exitErr instead of explicit error message and exitWith”) introduced wrong syntax (missing '$')- 29a3053 (“Remove unsafePerformIO usage”) changed the signature of...
Add two last missing opcodes
OpInstanceQuery was missing accidentally, whereas OpRestrictedCommandwas just recently added without Haskell definitions.
The patch also slightly improves the OpNodeQuery arbitrary generation.
Abstract query common opcode definitions
It would be even better if the opcodes would actually have all thesame definitions, until then we have two sets of definitions.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Remove remaining in-line parameters
… except one, and replace them with separately-defined ones inOpParams.
Replace hand-coded 'live' field of OpInstanceMigrate
This is the last inline-written field; I've done it separately fromthe last patch due to changes being needed in HTools (field changedtype).
Remove read instances from our Haskell code
It turns out that optimising 'read' derived instances (via -O) forcomplex data types (like OpCode, or the various objects) can be slowto very slow. Disabling such instances results in (time make$all_our_haskell_binaries) large compile-time savings and also smaller...
Complete the instance OpCodes and parameters
Only the original instance opcodes (used in htools) are leftnon-converted to only parameter style; they'll be cleaned up later,once the htools codebase itself migrates to safer types.
Add Group, OS and Backup opcodes
This also corrects a docstring in OpBackupExport on the Python side.
Complete the Tag opcodes and fix OpTestDelay missing param
The regexp in OpTagsSearch is loaded as is, without testing forvalidity; the rest of the patch is trivial.
Complete the Test opcodes
This adds the OpTestAllocator, OpTestJqueue and OpTestDummyopcodes. The OpTestAllocator seems to need some cleanup (on the Pythonside), for now we implement it as is. As for the other two, while notused in production, we should have full coverage for them as well....
Move NICMode from Objects.hs to Types.hs
Also add some unittests for this type.
Add types, parameters and the opcodes for networks
This completes the last missing opcode group. The only difficulty waswith the ip addresses, where we used simple strings to represent themand (for IPv4) a few helpers to generate arbitrary instances;otherwise, the patch is trivial....
Add types and parameters for OpInstanceCreate
This is a "big" opcode, so sending it separately.
A few types needed changing, and a few parameters were renamed to makeit more clear which are cluster-level and which are instance-levelparameters.
Move FileDriver from Objects to Types
Another base type that we need in opcodes as well.
Small improvements to the DRBD parser and tests
Implement the correct handling of numbers without commas
commaInt now recognizes only the first 3 digits for numbers without commas.
It was erroneously recognizing numbers of any size before the first comma.
A few more hlint fixes
I'm ignoring the use of Control.Arrow.*** since it's a non-obviousfunction, and we're not widely using Arrows in the code base, so Ithink switching to it would make the code less readable to mostpeople.
Bulk add of remaining cluster/node opcodes
These are tested for equivalence with the Python code using existingtests. These tests are not perfect - for many "TDict" we simply sendempty dicts, as we don't have a way to generate proper data, but thetests did catch many trivial errors during writing of this patch...
Convert a few existing opcode parameters to safer types
This does a partial conversion of existing opcodes to the newnon-empty string type, which is needed so that we don't add even moreopcodes without this safety net (and to have consistent "target_node",...
Add many more opcode parameters
This is a bulk add of all node parameters needed for node and clusteropcodes. The parameters are defined with a few helper functions inthis module, and, opposite from opcodes.py, there won't be any (new)opcode attributes declared in-line (without a separate field...
Add two more basic types
These are needed for completing the node opcode parameters.
Move StorageType from RPC.hs to Types.hs
This fixes a FIXME actually recommending this move :)
Split 'Query.Language.ItemType' in two sub-types
The QR_VIA_OP/QR_VIA_LUXI types in Python are using yet anothervalidation mode: QR_VIA_OP is the base type, and QR_VIA_LUXI extendsit (when doing luxi queries). But on the wire they have the samerepresentation....
Move the 'Hypervisor' type from Objects to Types
This is a very basic type and 'Objects' is a heavy-weight module. Bymoving it to 'types' we simplify (in the future) the import chains.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Adeodato Simo <dato@google.com>
Generalise the JSON "Container" type
Currently, we have some types that we kept as dictionaries in theobjects representation (due to inconsistent behaviour, or othertechnical reasons). This should be improved in the future, but in themeantime we can improve the Container type by allowing its keys to be...
Add more basic Ganeti types
These are needed both in opcodes and (potentially) outside, so we addthem in the main Types module.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add more basic validation types
This mirrors the ht.py types PositiveInt, NonNegative, etc., exceptthat they work at a more generic level (any numeric type, respectivelyany non-empty list).
Add parser for DRBD /proc file
Properly log errors when setting up daemon FDs
While writing the pipe-based reporting and trying various ways tobreak the startup, I fought for a while trying to understand why errorreporting was different when running the daemon as a user (with norights). It turns out that setupDaemonFDs wants to open the log file...
Switch opcode data type from normal to record constructors
Currently, the OpCode definitions are using normal constructors:
data OpCode = OpTestDelay Double Bool [String] | OpInstanceFailover String Bool (Maybe String) …
While this works for a few opcodes, it becomes unwieldy when dealing...
Fix docstrings for the Filter type
While looking at the opcode docs and clicking accidentally on thefilter type, I saw that the haddock formatting is broken due tonon-escaped use of special chars. Let's convert the ascii-likeformatting to haddock, and have nicer apidoc....
Split OpCode.hs and add module for opcode parameters
Due to TemplateHaskell stage restrictions, we can't define parametersin the same module as we're using them for TH, so we have to defineall module parameters in a separate module.
This patch therefore splits OpCodes.hs in two, adding that module and...
Create a new Ganeti/Types.hs module
There are already three cases where we copied type definitions betweenthe htools-specific types into the main ganeti code. Let's stop doingthis ☺ and create a common types module that holds these.
Note that there already exists BasicTypes.hs, but that refers to very...
Cleanup THH function use from built module namespace
Currently, THH.hs "injects" into the built code names of libraryfunctions like Text.JSON.makeObj, Ganeti.JSON.fromObj, etc. builtdirectly from strings, via (e.g.)
varE (mkName "makeObj")
This means that the "makeObj" name must exist in the target module,...
Expose changing job priority via LUXI
A new LUXI request is added, in both Python and Haskell.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix two hlint warnings
Sorry, I broke lint again :), by introducing two sub-standard changes.
Additionally, this silences an older existing warning that onlytriggers with some versions of hlint (e.g. 1.8.28 which is present inWheezy).
Merge branch 'devel-2.6' into master
Conflicts (both trivial): htools/Ganeti/Daemon.hs (_writePidFile rename)...
Merge branch 'stable-2.6' into devel-2.6
Fix PID file writing in Haskell daemons
Currently, the code uses createFile, which has the effect of alwaystruncating the file. This is bad, as the content of the PID file iswiped even when we wouldn't be able to lock it!
We switch to openFd (createFile is just a wrapper over that), and we...
Remove unsafePerformIO usage
We need to change a few things, most importantly CLI options defaults,but otherwise we already used the path to files from functions whichwere already in the I/O monad, so we don't have to change much of thecode flow.
Additionally, Path.hs now has an explicit export list, to stop...
Change keyToFilename/readSSconfFile interaction
Currently, keyToFilename uses itself the default config path. In thelight of making that an function in the IO monad, let's remove the"default" path functionality from this function and make it alwaysrequire the config path; its caller, readSSConfFile, can then do the...
Change type of program options to 'IO [Options]'
Some options have defaults that depend on the environment, and wecould handle these in two ways:
- use a place-holder value (e.g. data X a = Default | Custom a) that is later read from the environment- move the options list to IO monad, where it can read the...
Two small logging improvements
The "starting" message is changed to match the Python one, and in casethe preparation fails, we also log the error (beside printing it onstderr or writing it to the error reporting pipe), as at this timelogging is usually set up....
Annotate confd startup/luxi binding error
The luxi binding is now annotated for better readability.
Switch writing of PID file to new reporting style
Currently, the writing of the PID file uses a "standard" errorreporting: catch exception, transform it into a 'Result' type, leavehandling of that to the caller. However, for daemon startup, weactually want exceptions to be propagated up until the handler which...
Move and rename BasicTypes.annotateIOError
It turns out that annotateIOError already exists in the standardlibrary, with a different purpose (this made me waste 10 minutestrying to understand why the types were wrong…). Let's move this outof BasicTypes, since it's used only in Daemon.hs, and also rename it....
Switch daemon startup to pipe-based error reporting
This changes the daemon reporting error to the same mechanism as inPython: use a pipe which is written to from the forked children.
Move preparation steps of confd in prepMain
This does the address parsing earlier, before we fork, forbetter/faster error reporting.
Split queryd run into prepare and exec
This will help with the general daemon split of prepare/run, and flagerrors earlier in the startup.
Convert tag objects to a safer type
Currently, we keep information about the "target" of a tag operationin a data type similar to (TagKind, Maybe String). This is unsafe, asnothing (at the type level) prevents us from accidentally having(TagCluster, Just "instance1.example.com"), or (TagInstance, Nothing)....
Switch Luxi TH code from simple to custom fields
This is needed so that we have more flexibility in generating Luxiserialisation code (deserialisation is still custom). Also, onlyexceptions are now using the 'simple' field types, so we might be ablelater to convert and remove that TH code as well....
Make THH:genSaveOpCode a bit more general
This can be improved, by taking all hardcoded names as parameters, toserve as a more-general "build save clause for a multi-constructordata type". I'm not renaming the function as well, since I don't knowexactly how much we can abstract later....
Rename Confd.hs to Confd/Types.hs
This should be the last module rename, promise!
We rename this to conform to the other hierarchies (e.g. Query), andto not have both Confd.hs and Confd/*.hs.
OpCodes.hs: add OpTagsSet and OpTagsDel
Also, adjust comment to $(genOpCode) block to avoid repetition of"only".
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: move TagObject from Luxi.hs to OpCodes.hs
In addition to ReqQueryTags in Luxi.hs, the TagObject ADT is alsorequired for the "kind" attribute of OpTagsSet and OpTagsDel, whichare coming to OpCodes.hs next. Hence, we move TagObject there, andadjust imports accordingly....
Implement base checkFn/prepFn/execFn model for daemons
This mirrors the code in the Python code base, and is required forclean error reporting during startup.
This patch implements the basic infrastructure; the confd daemon isnot yet modified to take advantage of this, just the types are...
One more ghc 7.6 fix
This is only in master, so needed to be fixed separately.
Merge branch 'devel-2.6' into submit
Conflicts: Makefile.am (reordering, fixed) htools/Ganeti/Confd/Server.hs (hlint fixes on master) htools/Ganeti/Daemon.hs (hlint)...
Fix compatibility with newer Haskell libraries
This small patch fixes compatibility with a few newer Haskell libraries:
- base 4.6, included with ghc 7.6, removed the deprecated 'catch' function from Prelude, so our "import Prelude hiding (catch)" is now...
Add test for Luxi calls consistency hs/py
This tests that the same Luxi calls are defined in Python andHaskell. It doesn't test yet that their serialisation is correctthough.
Convert Luxi results to Ganeti errors
This a bit too complex patch converts the result of Luxi calls(submitJob, query*, etc.) from Result to ErrorResult. It thenimmediately revers this in the HTools/Backend/Luxi module, where wedon't need necessarily the full error type (just a nice error...
Move htools backends to a separate directory
Five modules under the HTools/ directories are backendimplementations, so let's move them to a separate directory, to moreclearly show the hierarchy. I wanted to do this for a while, butmerging between branches is always an issue, so let's do it know since...
Fix X509CertError definition in Haskell codebase
Thanks Dato for catching this.
Fix a few issues found by newer hlint
Testing with a newer hlint found a few minor issues; but all are real,valid recommendations:
- don't use "if cond then f x else f y", but "f (if cond then x else y)" - "if a then b else True" is equivalent to the simpler "not a || b"...
Add support for optional fields with null serialised
This follows a conversation we had for how to deal withoptional-but-required fields in JSON serialisations: fields which areoptional (can be either a given type or 'null'), but where the 'null'value is required. There are just a few of these in the Python code,...
Errors.hs: improve field names for ConfigVersionMismatch
Change {exp,act}Code to {exp,act}Ver, which gives a better idea thatthe integer fields represent version numbers.
Also:
- errors.py: update OpPrereqError's docstring to note that an error code is always expected as the second argument (it was previously...
THH.hs: delete isOptional, no longer used
The isOptional function is no longer used after a150585 (“Convertopcode TH code to the use of Field type”).
Convert query path from string errors to GanetiException
This patch converts all the call paths from 'Result' (which containsjust string errors) to 'ErrorResult', which holdsGanetiException-encoded errors. We can now return properOpPrereq/OpExec errors to the clients of the luxi/query socket....
Add exception utility functions
In Python, formatError also returns the exit code, but I find thatsplitting them leads to clearer code.
Add an Errors module mirroring the Python one
As described in the module doc string, while writing this it dawnedupon me that we're mixing all errors together into a single hierarchy(well, type on the Haskell side), which is not good. Some errors areused purely within noded, some in the CLI frontends, etc. so these...
Abstract Luxi template functionality
These are almost generic, so let's change the signatures a bit a makethem fully so.