Rework/enhance original htools opcodes
The original htools opcodes were minimalistic and not 1:1 equivalentwith the Python ones. Let's add all missing fields and, since wechanged the order, switch to more readable record syntax for buildingthe opcodes....
Add a few missing fields from opcodes
Due to manual conversion, a few fields were missing from theconversion, but as they were optional our type equivalence checkingdidn't detect this.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix generation of clusters via 'genEmptyCluster'
The current uniqueness algorithm (generate random node names, suffixthem with node index) is actually wrong: a node named "21" at index 5will end up with the same name as a node named "2" at position 15....
Add unit tests for the JSON serialization of DRBD status
The serialization itself is done by Text.JSON, so the tests deal with checkingthat Text.JSON objects are created correctly from the DRBD parser datastructures.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Add JSON serialization capabilities for DRBD status
The serialized JSON is not a 1:1 dump of the data structurespopulated by the parser. This is done intentionally, with theaim of producing a more stable and more meaningful output tobe used by the (future) monitoring agent and stand-alone data...
Reduce hostname length in tests and opcode py_compat count
This improves Issue 325 - new runtime and memory consumption is about1/10 compared to before.
Add generator for an instance on a node list
Given a node list in input, we get an instance that had nodes in it.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add generator for list of arbitrary instances
This takes an instance generator and produces a possibly empty list ofinstances.
Add generator for list of arbitrary nodes
This generates a minimum of one node, because legal clusters never havezero nodes.
Create a genInstanceMaybeBiggerThan function
This is used only once when testing Cluster.hs, but having it abstractedclarifies there what that call is about, makes that test shorter, andallows us to better do refactoring of the main genInstanceSmallerThan...
Add explicit test for wrong OpCode arbitrary instances
I've been bitten a couple of times with arbitrary opcodes working onUTF-8 locale, but failing on buildbot (ASCII). So let's add anexplicit test that checks always (even with UTF-8) for correctarbitrary values, showing explicitly which opcodes fail....
Remove read instances from our Haskell code
It turns out that optimising 'read' derived instances (via -O) forcomplex data types (like OpCode, or the various objects) can be slowto very slow. Disabling such instances results in (time make$all_our_haskell_binaries) large compile-time savings and also smaller...
Remove remaining in-line parameters
… except one, and replace them with separately-defined ones inOpParams.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Abstract query common opcode definitions
It would be even better if the opcodes would actually have all thesame definitions, until then we have two sets of definitions.
Make QuickCheck generator names more uniform
Due to lack of attention, we have two styles for generators ofarbitrary values: get* and gen* (e.g. getFQDN and genDiskIndices). Inorder to make this more obvious that we deal with a function in theGen monad, let's rename all get* functions to gen*....
Enable equivalence checks of opcode list
This patch enhances the opcode list checks - instead of spawning aPython interpreter to display the opcode list, we export it staticallyin Constants.hs via a slight convert-constants change.
Furthermore, since we now have opcode parity, we enable full opcode...
Add two last missing opcodes
OpInstanceQuery was missing accidentally, whereas OpRestrictedCommandwas just recently added without Haskell definitions.
The patch also slightly improves the OpNodeQuery arbitrary generation.
Signed-off-by: Iustin Pop <iustin@google.com>...
Add types, parameters and the opcodes for networks
This completes the last missing opcode group. The only difficulty waswith the ip addresses, where we used simple strings to represent themand (for IPv4) a few helpers to generate arbitrary instances;otherwise, the patch is trivial....
Move NICMode from Objects.hs to Types.hs
Also add some unittests for this type.
Complete the Test opcodes
This adds the OpTestAllocator, OpTestJqueue and OpTestDummyopcodes. The OpTestAllocator seems to need some cleanup (on the Pythonside), for now we implement it as is. As for the other two, while notused in production, we should have full coverage for them as well....
Complete the Tag opcodes and fix OpTestDelay missing param
The regexp in OpTagsSearch is loaded as is, without testing forvalidity; the rest of the patch is trivial.
Add Group, OS and Backup opcodes
This also corrects a docstring in OpBackupExport on the Python side.
Complete the instance OpCodes and parameters
Only the original instance opcodes (used in htools) are leftnon-converted to only parameter style; they'll be cleaned up later,once the htools codebase itself migrates to safer types.
Add types and parameters for OpInstanceCreate
This is a "big" opcode, so sending it separately.
A few types needed changing, and a few parameters were renamed to makeit more clear which are cluster-level and which are instance-levelparameters.
Move FileDriver from Objects to Types
Another base type that we need in opcodes as well.
Small improvements to the DRBD parser and tests
Fix wrong edge case in a QuickChek Drbd property
The bug was in the test itself, not in the tested code.
Also, fixed a line longer than 80 characters in the same file.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
QuickCheck Tests for the commaIntParser, part of DRBD Parser
These new tests substitute the HTests, as by the suggestionreceived after submitting the previous patch.
Add two more basic types
These are needed for completing the node opcode parameters.
Convert a few existing opcode parameters to safer types
This does a partial conversion of existing opcodes to the newnon-empty string type, which is needed so that we don't add even moreopcodes without this safety net (and to have consistent "target_node",...
Bulk add of remaining cluster/node opcodes
These are tested for equivalence with the Python code using existingtests. These tests are not perfect - for many "TDict" we simply sendempty dicts, as we don't have a way to generate proper data, but thetests did catch many trivial errors during writing of this patch...
Fix OpCode arbitrary generation w.r.t. non-ASCII chars
Since we generate arbitrary strings and then we send them over a pipeto Python, we can (and will) generate non-ascii chars, which meanswhen run in a non-UTF8 locale, this will break with ghc 6.12 and...
A few more hlint fixes
I'm ignoring the use of Control.Arrow.*** since it's a non-obviousfunction, and we're not widely using Arrows in the code base, so Ithink switching to it would make the code less readable to mostpeople.
Add an arbitrary set generator helper
This makes it easy to generate sets of "things" that don't even needto have arbitrary instances, but which have bounded/enum instances.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Adeodato Simo <dato@google.com>
Add more basic Ganeti types
These are needed both in opcodes and (potentially) outside, so we addthem in the main Types module.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Generalise the JSON "Container" type
Currently, we have some types that we kept as dictionaries in theobjects representation (due to inconsistent behaviour, or othertechnical reasons). This should be improved in the future, but in themeantime we can improve the Container type by allowing its keys to be...
Move the 'Hypervisor' type from Objects to Types
This is a very basic type and 'Objects' is a heavy-weight module. Bymoving it to 'types' we simplify (in the future) the import chains.
Split 'Query.Language.ItemType' in two sub-types
The QR_VIA_OP/QR_VIA_LUXI types in Python are using yet anothervalidation mode: QR_VIA_OP is the base type, and QR_VIA_LUXI extendsit (when doing luxi queries). But on the wire they have the samerepresentation....
Move StorageType from RPC.hs to Types.hs
This fixes a FIXME actually recommending this move :)
Fix some more hlint warnings
Sorry…
Add more basic validation types
This mirrors the ht.py types PositiveInt, NonNegative, etc., exceptthat they work at a more generic level (any numeric type, respectivelyany non-empty list).
Fix a hlint warning
This is yet-another version-specific warning (fixed in later hlintversions), but it's better to keep it clean.
Add DRBD parser unit tests
This adds tests that existing test files can be parsed by the Haskellparser as well, plus one new test file.
Simplify a few test helpers
Use more monadic combinators instead of explicit code.
Create a new Ganeti/Types.hs module
There are already three cases where we copied type definitions betweenthe htools-specific types into the main ganeti code. Let's stop doingthis ☺ and create a common types module that holds these.
Note that there already exists BasicTypes.hs, but that refers to very...
Add Utility functions for loading data from test files
They mimic their python counterparts.
Improved TestHelper module docstring
Cleanup THH function use from built module namespace
Currently, THH.hs "injects" into the built code names of libraryfunctions like Text.JSON.makeObj, Ganeti.JSON.fromObj, etc. builtdirectly from strings, via (e.g.)
varE (mkName "makeObj")
This means that the "makeObj" name must exist in the target module,...
Switch Attoparsec/unicode test from QC to HUnit
This is not a quickcheck property, since it doesn't have anyvariable/arbitrary inputs. So let's make it a test case, and shorten abit the name.
Also, sorry for not catching this in the review.
Expose changing job priority via LUXI
A new LUXI request is added, in both Python and Haskell.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Automatically enable version-dependent GHC flags
Some GHC flags are very useful, but only appear in more recent GHCversions. To support the use of such flags while still supportingolder compilers, let's add conditional checks and enabling based onthe results....
Added attoparsec unit test for Unicode parsing
Attoparsec is known to have had issues with parsing non-ASCII strings.This test makes sure that parsing of Unicode characters works fine.
Signed-off-by: Michele Tartara <mtartara@google.com>[iustin: small doc string fixes]...
Fix two hlint warnings
Sorry, I broke lint again :), by introducing two sub-standard changes.
Additionally, this silences an older existing warning that onlytriggers with some versions of hlint (e.g. 1.8.28 which is present inWheezy).
Remove unsafePerformIO usage
We need to change a few things, most importantly CLI options defaults,but otherwise we already used the path to files from functions whichwere already in the I/O monad, so we don't have to change much of thecode flow.
Additionally, Path.hs now has an explicit export list, to stop...
Change keyToFilename/readSSconfFile interaction
Currently, keyToFilename uses itself the default config path. In thelight of making that an function in the IO monad, let's remove the"default" path functionality from this function and make it alwaysrequire the config path; its caller, readSSConfFile, can then do the...
Change type of program options to 'IO [Options]'
Some options have defaults that depend on the environment, and wecould handle these in two ways:
- use a place-holder value (e.g. data X a = Default | Custom a) that is later read from the environment- move the options list to IO monad, where it can read the...
Convert tag objects to a safer type
Currently, we keep information about the "target" of a tag operationin a data type similar to (TagKind, Maybe String). This is unsafe, asnothing (at the type level) prevents us from accidentally having(TagCluster, Just "instance1.example.com"), or (TagInstance, Nothing)....
Fix display of results in ==? operator
Doh, this is exactly the opposite of what we wanted… good thing notest failed so far :)
Rename Confd.hs to Confd/Types.hs
This should be the last module rename, promise!
We rename this to conform to the other hierarchies (e.g. Query), andto not have both Confd.hs and Confd/*.hs.
OpCodes.hs: add OpTagsSet and OpTagsDel
Also, adjust comment to $(genOpCode) block to avoid repetition of"only".
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: move TagObject from Luxi.hs to OpCodes.hs
In addition to ReqQueryTags in Luxi.hs, the TagObject ADT is alsorequired for the "kind" attribute of OpTagsSet and OpTagsDel, whichare coming to OpCodes.hs next. Hence, we move TagObject there, andadjust imports accordingly....
Merge branch 'devel-2.6' into submit
Conflicts: Makefile.am (reordering, fixed) htools/Ganeti/Confd/Server.hs (hlint fixes on master) htools/Ganeti/Daemon.hs (hlint)...
Add test for Luxi calls consistency hs/py
This tests that the same Luxi calls are defined in Python andHaskell. It doesn't test yet that their serialisation is correctthough.
Move htools backends to a separate directory
Five modules under the HTools/ directories are backendimplementations, so let's move them to a separate directory, to moreclearly show the hierarchy. I wanted to do this for a while, butmerging between branches is always an issue, so let's do it know since...
Fix lint issue in Test/…/THH.hs
Sorry!
Fix a few issues found by newer hlint
Testing with a newer hlint found a few minor issues; but all are real,valid recommendations:
- don't use "if cond then f x else f y", but "f (if cond then x else y)" - "if a then b else True" is equivalent to the simpler "not a || b"...
Add a simple unittest for THH code
This is very THH specific, and applies to all serialisations generatedby THH, so I'm adding it in its own module.
Probably we should add some more generic tests, but in general THHcode is tested by the various definitions; this new field type however...
Convert query path from string errors to GanetiException
This patch converts all the call paths from 'Result' (which containsjust string errors) to 'ErrorResult', which holdsGanetiException-encoded errors. We can now return properOpPrereq/OpExec errors to the clients of the luxi/query socket....
Add an Errors module mirroring the Python one
As described in the module doc string, while writing this it dawnedupon me that we're mixing all errors together into a single hierarchy(well, type on the Haskell side), which is not good. Some errors areused purely within noded, some in the CLI frontends, etc. so these...
Remove multiple uses of '.&&.' with conjoin
This is just a bit of cleanup. The (.&&.) operator is internally just:
a .&& b = conjoin [a, b]
so let's replace 'a .&&. b .&&. c .&&. d' directly with 'conjoin [a,b, c, d]'.
Improve message for (==?) operator
After seeing how nice HUnit formats the error message on failed'assertEqual', I think we can do better with ==?. Currently it says(on one line): "Expected equality, but 1 /= 2".
This patch changes the code to format it similar to HUnit:...
Annotated inequality operator for unit test properties
This includes: * The operator (/=?), which checks for inequality and prints an error message if it encounters equality. (Basically the negation of the (==?) operator). * Application of this operator in the test property...
Fix setting of 'failN1' flag for corner case
This patch includes:
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
A few unittests improvements
Small simplifications of other unit tests using the (==?)operator when possible, and typo fixes.
Remove custom OpResult type/monad
Since we now have the GeneralResult as a multi-purpose monad, we canremove the custom OpResult monad, and just use 'GeneralResultFailMode' as our type. This allows removal of a few bits ofspecialised infrastructure, relying instead on the generic one....
Use exitErr instead of explicit error message and exitWith
Furthermore, a few messages have their capitalisation changed (fixed).
Signed-off-by: Helga Velroyen <helgav@google.com>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Generalise the Result type
Currently, our error monad—Result—has a plain string error type. Thisis not good, as we don't have structured errors, we can't pass backproper error information to Python code, etc.
To solve this, we generalise this type as 'GenericResult a', and make...
Merge branch 'devel-2.6' into master
Cleanup HTools.Types/BasicTypes imports
Before we reorganised the source tree, the 'Result' type was exportedfrom HTools/Types.hs. This changed during the reorg, but at that timewe didn't change the exports; instead, we kept re-exporting it fromthe old module for compatibility....
Cleanup/expand the filter/query tests
This patch cleans up duplicate code in Test.Ganeti.Query.Filter andthen adds a test for names consistency with Python's code behaviour(stable ordering for simple filters and otherwise niceSort'edordering).
Add functionality for checking validity of names
This replicates in the Haskell Query2 implementation the behaviour ofthe Python code: if a "simple" filter is passed (one that containsonly Or aggregators and EQ binary ops on the name field), then anfailure is flagged if the given values are not known....
Switch ordering of names on query to niceSort
This makes the "all" names queries consistent with the Pythonresults. The change requires updating the unittests, at which point aduplicate error message is simplified.
Add a makeSimpleFilter function
And associated unittests. This will be needed for classic-stylequeries.
(htest) Text.hs: fix typo in comment
Add an Utils.NiceSort() equivalent
This patch adds a NiceSort equivalent and the corresponding unittest(partially copied from Python unittest). The difference between thePython version and this one is that this implementation doesn't useregular expressions, and as such it doesn't have the 8-groups...
Define the actual arguments that are supported by programs
This defines the arguments supported and then modifies the--help-completion output to include them too.
Remove qualified import of Utils.hs into its test module
This is a leftover from the times when we had a single, huge testmodule; nowadays it's only an annoyance.
Rename Ganeti/HTools/Utils.hs to Ganeti/Utils.hs
This is, I believe, the last non-htools specific file that still livedin the htools directory; it's already widely used in non-htools code,so let's move it before we add more functionality to this module....
Add live parameter to query
The tests we currently have assume, that all the data required forrunning the query is available - once we add live data, this will nolonger be the case.
This patch adds boolean parameter to query function, which tells itwhether to ignore live parameters gathering....
Tests for NodeGroup query
This adds tests similar to those used for node query. For now theprop_queryGroup_noUnknown is disabled and commented out, as it isfasifiable with ndparams and ipolicy. It may be removed or fixedlater on.
Also, prop_queryGrooup_types has one less property checked - it is...
Remove generic options from individual programs
Currently, we test and require that each individual program (hbal,etc.) defines/supports the generic options (currently --help and--version). Even with the test, this is not optimal, since it requireschanges in many places whenever we modify the list of generic options,...
Annotate options with completion information
This is a quite boring patch, just adding annotation information toall existing options. Some of the annotations are not very good; butwe don't have support for more precise completion inbuild-bash-completion, so this is good enough....
Improve the prop_Alloc_sane test to detect mis-allocations
Currently, this just checks that a cluster cannot be rebalanced aftera single instance allocation. However, we can also test whether theallocation decision computed a correct new cluster score, by checking...
Add new test for checking multi-allocations
This test expands the "single-alloc-no-rebalance" by allocating a fewinstances on a small cluster, and ensuring that after we allocate allof them, either we can't rebalance or if we rebalance the scoreimprovement is very small....
Improve the `CanTieredAlloc' test
Currently, this test is very slow. Upon investigation, this is due tohow `tieredAlloc' works:
- tries to allocate one instance- if failed, shrink the instance by the "most failed" resource- restart
In this algorithm, if the "most failed" resource is e.g. memory, and...
Improve the `AllocPolicy' test
This test has a few deficiencies, which this patch addresses:
- using arbitrary 1 or 2 node count for allocation is obsolete, nowadays we need to use a number appropriate for the instance's disk template (and we should remove that parameter…)...
Fix arbitrary ConfigData object generation
The Cluster object, as it is defined right now, has many '[String]'members, which means that in a standard arbitrary generator these willbecome very big, which is the reason for the current slowness of thetest 'Config_serialisation'....
Remove the slow/fast tests functionality
Since the recent commits improved the speed of the two "slow" testgroups to regular test speed, we can remove this kludge and simplifysignificantly our test runner, yay!
Simplify a bit more the test harness
We can build the test groups directly in the `testSuite' helper,instead of doing it (much later) in the test harness.
Further hlint fixes
Commit 2cdaf22, “Re-enable standard hlint warnings”, got it almostright. The only problem is that (confusingly) the default set of hintsis not in HLint.Default, but in HLint.HLint (it includes Default andsome built-ins).
After changing the lint file to correctly include the defaults, we had...