Make node groups' networks field a dict of nicparams
The networks field of the node groups class is a dictionaryof nic params and not a list of networks. This should fixthe broken QA.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add hroller htools personality
This is a new personality that for the moment doesn't do anything.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Renaming fam -> net_family on test network generation
Small renaming to make the test more readable.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Serialization compatibility test for node groups
This creates a number of node groups with up to 3 networks,serializes them and compares them with their python equvalents.
Add a network fields to NIC params and node group type
This adds a network field to the NIC params and a list of networksto the node group type. It adjusts the nodegroup's unit tests accordingly.
Add support for job queries in hconfd
This adds support for job queries, including (basic) unit-tests.
I've tested this for memory and cpu usage as follows:
- 3600 jobs (live queue): - via masterd, default: ~1.1s (masterd: ~60MB ram) - via confd, default: ~1.1s (hconfd: ~25MB ram)...
Change makeSimpleFilter accept NumericValues as well
This is required for QueryJobs implementation; we changemakeSimpleFilter to support both string and integers as names.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add a read-only job queue module
This patch adds implementation for a read-only job queue module,together with "full" test (as full as can be in a lazy language…).
One note about the behaviour of the job queue is the handling ofopcodes that fail validation: the 'input' opcode actually is a...
Implement support for QFF_SPLIT_TIMESTAMP
I am very very unhappy with this patch. We have to do this, in orderto be compatible with the Python code and behaviour, but there's lotsof duplication; both the Python and the Haskell code will need to becleaned up and simplified (we don't need QFF_* at all)....
Remove superfluous typing to Int
Forgot to fix that after the review of patch"Haskell/python compatibility test for networks".
Extend job ID parsing tests
Since this is a type that behaves non-standard, let's add some moretest for "wrong" input values.
Add a few tests for OpParams types
As OpParams definitions are used at Template Haskell type, they don'tget any coverage (although the values defined by them are actuallyused in unittests). However, we can at least test some of thefunctions defined in the module....
Add more mon-collector shell tests
These test the newly-changed behaviour, and fix an inconsistency inthe hs-check target (versus hs-coverage).
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Add description to personality definitions
This allows usage information to display nicer help (like in Python).
Allow shutting down offline instance
If an instance is offline we definitely shouldn't start it up.But shutting it down, should it be up by mistake is not "that" bad.Still, we only allow it with --force, as it still performs an action onan instance we shouldn't touch. This should make everybody happy....
Haskell/python compatibility test for networks
This patch contains the HUnit test that checks the compatibilityof Haskell-generated networks with the python code. For that thegeneration of test instances of networks was enhanced to meetthe validation steps of the python implementation. Also, so far...
Split and extend Confd types tests
Some confd types were not tested for serialisation, so let's movethese (type-related) tests to their own module and extend them.
Reduce logging level in tests
This allows tests to execute IO actions which have logging as a sideeffect, without polluting the stderr too much.
A better solution would be that we have fine-grained control overloggers, so that tests can run with their own logging, etc. etc.....
Generalise the test helper 'genUniquesList'
Instead of always using "arbitrary", make it require a generator (thatcan then use more complex rules for building the list elements), andalso convert it to use a temporary set instead of list membership.
Signed-off-by: Iustin Pop <iustin@google.com>...
Implement opcode summary support
This implements in the Haskell codebase the opcode summary. As opposedto Python, we always use custom code for formatting, since we don'twant to use dynamic attribute lookup.
To test this properly, we need to change MetaOpCode to record-syntax,...
Add a type for the log type in job/opcode logs
Also fixes indentation for OP_STATUS.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Improve a few Graph test properties
Return type is changed from Property to Bool, and the ==? True at theend is dropped.
Add shelltests for the mon-collector
This commit adds shelltests for the mon-collector binary and for the DRBDdata collector.
Also, it fixes a small bug in the DRBD parser found thanks to the tests.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Remove usage of reserved key word 'family'
An old hlint version (1.8.28) was complaining about theword 'family' being used in the network tests. Thanks toan epiphany of Iustin this was due to that it is a reservedkey word.
Signed-off-by: Helga Velroyen <helgav@google.com>...
Network and address pool (again)
Implementation of the network objects and address pool. Functionalityas in the reverted commit b9a616e132af9bba718d2da8c94eeec7af886814, butnow using only the vector library and not the bit-vector library.Tested with vector library version 0.9 and 0.10., which are...
htest/Types.hs: check ordering of some ADTs
In some ADT, ordering of constructors is important because a certainmeaning is attached to ordinality (e.g., "clusters with lesser AllocPolicyare preferred").
To ensure constructor order is not accidentally changed, this commit...
htools: Add missing parameter to OpInstanceMultiAlloc
Somehow this went missing in commit 1f1188c.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
cmdlib: Opportunistic locking on instance creation
Adds a new parameter to “OpInstanceCreate” and “OpInstanceMultiAlloc” touse opportunistic locks.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Revert "Network and address pool"
This reverts commit b9a616e132af9bba718d2da8c94eeec7af886814, whichdepends on the "bitvec" library. We need to investigate a bit further thatdependency, as it in turns depends on "vector" 0.9.1 or below, since0.10.* removed support for Data.Vector.Unboxed.Safe which it uses....
Network and address pool
Implementation of the network and address pool class inHaskell. Not complete yet. Includes unit tests that coverall functionality that is so far implemented.
make genMaybe more Just
There is a common conception that Just something is more worth thanNothing. So we're biasing our tests towards that. As such let's generateNothing fewer times, and Just subgen more times. The values were copiedfrom the "official" maybe generator....
Fix build breakage in Jobs.hs test code
Commit 3bdbe4b3 (“Jobs.hs: move OpStatus and JobStatus ADTs toTypes.hs”) removed the TemplateHaskell language pragma fromhtest/Test/Ganeti/Jobs.hs due to a hlint warning, but that is bad: itmeans the testSuite call is no longer interpreted as a splice, so it results in:...
Jobs.hs: move OpStatus and JobStatus ADTs to Types.hs
This leaves Ganeti/Jobs.hs and Test/Ganeti/Jobs.hs empty, but they're thetarget of a future move of some functions, so we leave them around, anddon't delete them, to avoid unnecessary delete/create diffs....
Add Ganeti.HTools.Graph
This module implements some algorithms on Data.Graph data structures.At the moment its main functionality is an LF-color implementation(greedy coloring in descending order of degree). There are also a fewextra functions to calculate the degree order, and convert the node to...
Add Dsatur implementation
Implement the Dsatur algorithm for Graph coloring. This also abstractsthe neighColors function into two subfunctions that this algorithm canreuse.
Ganeti/HTools/Graph Add isColorable
Check whether coloring on a given graph makes sense. This is the caseonly if there are no loops and the graph is undirected.
Add tests for verticesByDegree{Asc,Desc}
This brings our coverage of Graph.hs to 100%
HTools/Node: add mkNodeGraph function
This function helps treating node node problems as graph problems.It can transform a list of nodes plus a list of instances into a graphwhich uses the nodes as vertices, and instances as edges connecting them(as long as they have both a primary and a secondary node)...
Fix Dsatur and add Dcolor
Our Dsatur implementation was incorrect: while the paper defined thedegree of saturation of a vertex as the number of different colors it isadjacent to, we were using the number of colors, without consideringuniqueness. This effectively implemented a different algorithm, which is...
Add "proper coloring" unittest check
We have to check that for each edge its vertices have different colors.
This is very easy to do with a vertex-to-color map, but not so easy witha color-to-vertex one. Since all our coloring algorithms created avertex-to-color map behind the scenes and then converted it, we flip...
Add a negative type
This mirrors the positive one, and will be needed for relative jobIDs.
Add types and parameters for common opcode implementation
This will go into a separate type; this patch adds the neededunderlying types and parameters.
Add CommonOpParams and MetaOpCode types
This patch adds the "meta" opcode type and the common opparams. Compatibility tests with Python are changed to pass Metaopcodes.
Rework/enhance original htools opcodes
The original htools opcodes were minimalistic and not 1:1 equivalentwith the Python ones. Let's add all missing fields and, since wechanged the order, switch to more readable record syntax for buildingthe opcodes....
Add a few missing fields from opcodes
Due to manual conversion, a few fields were missing from theconversion, but as they were optional our type equivalence checkingdidn't detect this.
Fix Haskell OpNetworkAdd
Commit 213076f (“Fix locking in networks”) changed Python OpNetworkAddwithout corresponding Haskell definition changes.
Add test for opcode fields equivalence
As opposed to the existing test, which tests the type/serialisation offields, this one simply tests the equivalence of the list of fieldsfor each opcode.
Add type for finalised job statuses
For now, we don't need a pending job status type as well, so we'lldelay adding that until later.
Add a 'real' type for JobIds
Currently, the job ID is a simple type alias. This is suboptimal, asit means we can't use a custom JSON (or Arbitrary) instance for it.
The patch changes it into a newtype, and then a) simplifies somedeserialisation code and b) changes some more fields to this new type...
Fix generation of clusters via 'genEmptyCluster'
The current uniqueness algorithm (generate random node names, suffixthem with node index) is actually wrong: a node named "21" at index 5will end up with the same name as a node named "2" at position 15....
Add JSON serialization capabilities for DRBD status
The serialized JSON is not a 1:1 dump of the data structurespopulated by the parser. This is done intentionally, with theaim of producing a more stable and more meaningful output tobe used by the (future) monitoring agent and stand-alone data...
Add unit tests for the JSON serialization of DRBD status
The serialization itself is done by Text.JSON, so the tests deal with checkingthat Text.JSON objects are created correctly from the DRBD parser datastructures.
Reduce hostname length in tests and opcode py_compat count
This improves Issue 325 - new runtime and memory consumption is about1/10 compared to before.
Add generator for an instance on a node list
Given a node list in input, we get an instance that had nodes in it.
Add generator for list of arbitrary instances
This takes an instance generator and produces a possibly empty list ofinstances.
Add generator for list of arbitrary nodes
This generates a minimum of one node, because legal clusters never havezero nodes.
Create a genInstanceMaybeBiggerThan function
This is used only once when testing Cluster.hs, but having it abstractedclarifies there what that call is about, makes that test shorter, andallows us to better do refactoring of the main genInstanceSmallerThan...
Add two last missing opcodes
OpInstanceQuery was missing accidentally, whereas OpRestrictedCommandwas just recently added without Haskell definitions.
The patch also slightly improves the OpNodeQuery arbitrary generation.
Enable equivalence checks of opcode list
This patch enhances the opcode list checks - instead of spawning aPython interpreter to display the opcode list, we export it staticallyin Constants.hs via a slight convert-constants change.
Furthermore, since we now have opcode parity, we enable full opcode...
Make QuickCheck generator names more uniform
Due to lack of attention, we have two styles for generators ofarbitrary values: get* and gen* (e.g. getFQDN and genDiskIndices). Inorder to make this more obvious that we deal with a function in theGen monad, let's rename all get* functions to gen*....
Abstract query common opcode definitions
It would be even better if the opcodes would actually have all thesame definitions, until then we have two sets of definitions.
Remove remaining in-line parameters
… except one, and replace them with separately-defined ones inOpParams.
Remove read instances from our Haskell code
It turns out that optimising 'read' derived instances (via -O) forcomplex data types (like OpCode, or the various objects) can be slowto very slow. Disabling such instances results in (time make$all_our_haskell_binaries) large compile-time savings and also smaller...
Add explicit test for wrong OpCode arbitrary instances
I've been bitten a couple of times with arbitrary opcodes working onUTF-8 locale, but failing on buildbot (ASCII). So let's add anexplicit test that checks always (even with UTF-8) for correctarbitrary values, showing explicitly which opcodes fail....
Complete the instance OpCodes and parameters
Only the original instance opcodes (used in htools) are leftnon-converted to only parameter style; they'll be cleaned up later,once the htools codebase itself migrates to safer types.
Add Group, OS and Backup opcodes
This also corrects a docstring in OpBackupExport on the Python side.
Complete the Tag opcodes and fix OpTestDelay missing param
The regexp in OpTagsSearch is loaded as is, without testing forvalidity; the rest of the patch is trivial.
Complete the Test opcodes
This adds the OpTestAllocator, OpTestJqueue and OpTestDummyopcodes. The OpTestAllocator seems to need some cleanup (on the Pythonside), for now we implement it as is. As for the other two, while notused in production, we should have full coverage for them as well....
Move NICMode from Objects.hs to Types.hs
Also add some unittests for this type.
Add types, parameters and the opcodes for networks
This completes the last missing opcode group. The only difficulty waswith the ip addresses, where we used simple strings to represent themand (for IPv4) a few helpers to generate arbitrary instances;otherwise, the patch is trivial....
Add types and parameters for OpInstanceCreate
This is a "big" opcode, so sending it separately.
A few types needed changing, and a few parameters were renamed to makeit more clear which are cluster-level and which are instance-levelparameters.
Move FileDriver from Objects to Types
Another base type that we need in opcodes as well.
Small improvements to the DRBD parser and tests
Fix wrong edge case in a QuickChek Drbd property
The bug was in the test itself, not in the tested code.
Also, fixed a line longer than 80 characters in the same file.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
QuickCheck Tests for the commaIntParser, part of DRBD Parser
These new tests substitute the HTests, as by the suggestionreceived after submitting the previous patch.
A few more hlint fixes
I'm ignoring the use of Control.Arrow.*** since it's a non-obviousfunction, and we're not widely using Arrows in the code base, so Ithink switching to it would make the code less readable to mostpeople.
Fix OpCode arbitrary generation w.r.t. non-ASCII chars
Since we generate arbitrary strings and then we send them over a pipeto Python, we can (and will) generate non-ascii chars, which meanswhen run in a non-UTF8 locale, this will break with ghc 6.12 and...
Bulk add of remaining cluster/node opcodes
These are tested for equivalence with the Python code using existingtests. These tests are not perfect - for many "TDict" we simply sendempty dicts, as we don't have a way to generate proper data, but thetests did catch many trivial errors during writing of this patch...
Convert a few existing opcode parameters to safer types
This does a partial conversion of existing opcodes to the newnon-empty string type, which is needed so that we don't add even moreopcodes without this safety net (and to have consistent "target_node",...
Add two more basic types
These are needed for completing the node opcode parameters.
Add an arbitrary set generator helper
This makes it easy to generate sets of "things" that don't even needto have arbitrary instances, but which have bounded/enum instances.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Adeodato Simo <dato@google.com>
Move StorageType from RPC.hs to Types.hs
This fixes a FIXME actually recommending this move :)
Split 'Query.Language.ItemType' in two sub-types
The QR_VIA_OP/QR_VIA_LUXI types in Python are using yet anothervalidation mode: QR_VIA_OP is the base type, and QR_VIA_LUXI extendsit (when doing luxi queries). But on the wire they have the samerepresentation....
Move the 'Hypervisor' type from Objects to Types
This is a very basic type and 'Objects' is a heavy-weight module. Bymoving it to 'types' we simplify (in the future) the import chains.
Generalise the JSON "Container" type
Currently, we have some types that we kept as dictionaries in theobjects representation (due to inconsistent behaviour, or othertechnical reasons). This should be improved in the future, but in themeantime we can improve the Container type by allowing its keys to be...
Add more basic Ganeti types
These are needed both in opcodes and (potentially) outside, so we addthem in the main Types module.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix some more hlint warnings
Sorry…
Add more basic validation types
This mirrors the ht.py types PositiveInt, NonNegative, etc., exceptthat they work at a more generic level (any numeric type, respectivelyany non-empty list).
Fix a hlint warning
This is yet-another version-specific warning (fixed in later hlintversions), but it's better to keep it clean.
Add DRBD parser unit tests
This adds tests that existing test files can be parsed by the Haskellparser as well, plus one new test file.
Simplify a few test helpers
Use more monadic combinators instead of explicit code.
Create a new Ganeti/Types.hs module
There are already three cases where we copied type definitions betweenthe htools-specific types into the main ganeti code. Let's stop doingthis ☺ and create a common types module that holds these.
Note that there already exists BasicTypes.hs, but that refers to very...
Add Utility functions for loading data from test files
They mimic their python counterparts.
Improved TestHelper module docstring
Cleanup THH function use from built module namespace
Currently, THH.hs "injects" into the built code names of libraryfunctions like Text.JSON.makeObj, Ganeti.JSON.fromObj, etc. builtdirectly from strings, via (e.g.)
varE (mkName "makeObj")
This means that the "makeObj" name must exist in the target module,...
Switch Attoparsec/unicode test from QC to HUnit
This is not a quickcheck property, since it doesn't have anyvariable/arbitrary inputs. So let's make it a test case, and shorten abit the name.
Also, sorry for not catching this in the review.
Expose changing job priority via LUXI
A new LUXI request is added, in both Python and Haskell.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Automatically enable version-dependent GHC flags
Some GHC flags are very useful, but only appear in more recent GHCversions. To support the use of such flags while still supportingolder compilers, let's add conditional checks and enabling based onthe results....
Added attoparsec unit test for Unicode parsing
Attoparsec is known to have had issues with parsing non-ASCII strings.This test makes sure that parsing of Unicode characters works fine.
Signed-off-by: Michele Tartara <mtartara@google.com>[iustin: small doc string fixes]...
Fix two hlint warnings
Sorry, I broke lint again :), by introducing two sub-standard changes.
Additionally, this silences an older existing warning that onlytriggers with some versions of hlint (e.g. 1.8.28 which is present inWheezy).