History | View | Annotate | Download (66.4 kB)
Fix corner case in quick-check generated tests
Thanks to Agata Murawska, a random seed that generates bad test caseswas found ("seed was 1768143307 2118231514, test size 74"). This isdue to the fact that some nodes were generated with tMem=0, fMem=0,which means that pMem was NaN and thus the node was not comparing...
Fix prefix bug in Haskell ssconf implementation
This patch fixes a (brown-bag) bug in the ssconf implementation wherethe looked-for filenames were not prefixed with the ssconf fileprefix.
Additionally, a test for this is added (which fails without the fix)....
Add decoding of Luxi calls and unittests for LuxiOp
This patch adds a hand-coded decoder for LuxiCall arguments, as thedata-structure is not uniform enough for automated generation (evenfor the serialisation, we had to add hints for some fields,de-serialisation is even harder)....
Some unit tests improvements
In preparation for unit-testing the Luxi definitions, this improvesthe auto-generation of OpCodes: currently this uses unbounded plainStrings for names and fields, and this creates too big test sizes(e.g. when automatically generating job sets)....
Add a new JSON function
And its associated unittests.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
htools: Reflect the rename of spindle_usage to spindle_use
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add a helper function for mirrorType computations
We most always use `templateMirrorType . diskTemplate`, so let's add ahelper function for this.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Enable all disk templates in unit-tests
After the many patches implementing the shared storage functionalityin iallocator, balances, etc., and after the patches preparing theunit-tests to do the right thing for shared storage, we can finallyflip the bits that make instances have "random" disk templates as...
Fix Node_addPriFD test for arbitrary disk templates
Currently, this test assumes localStorageTemplates, so let's make surewe enforce that on the given instance
htools: Add the new ipolicy field
Also add the spindle usage to QC
htools: Adapt the Text backend to spindle usage
The old text files remain compatible with this change.
htools: Adding spindle usage to Instance
For now we just pass in "1" as spindle usage and adapt the backendlater.
Rework unit parsing
Due to how conversions were implemented previously, 1TB failed toparse on 32-bit, as we were overflowing during computation, eventhough the final result would fit easily.
This patch moves the parsing of the scaling factor to a separate...
Fix relocation test ClusterAllocRelocate
As for the evacuate tests, we require a mirrored disk template, butotherwise the test should work for both mirror types.
Additionally, we perform a simplification that was left as TODO.
Signed-off-by: Iustin Pop <iustin@google.com>...
Fix up Cluster nodeEvac unittests for shared storage
Currently, these presume that the instance being generated always isDRBD/relocatable (but won't be in the future), and also presume thatall relocation modes can be used (hint: not ☺).
So we add a couple of helper functions to restrict the used disk...
Fix a few tests that hard-coded the number of nodes
Instead of randomly generating the number of required nodes orhard-coding it, compute it from the disk template of theinstance. Otherwise tests will fail when the disk template is notDRBD8 and we ask for allocation on 2 nodes....
Fix test Node_addSec_idempotent
Another test that only works for DRBD8 instances.
Fix the Node_rMem test to specify DRBD8 templates
… since memory reservation only is used for this template.
Split the Node_addOffline test into pri/sec
Currently, this (single) test checks whether we can add an instance asprimary and secondary to a given node. This won't work for instancetemplates other than DRBD8, so we split it into:
- an add to primary test, which should work for all disk templates...
Do not rely on no-secondary for movable tests
Currently, the code in Loader.hs marks as un-movable any instanceswhich don't have a secondary. This is not valid for any shared storagetemplates, so let's remove that check completely. The initial state of...
Fixup a Node unittest
This fixes the prop_Node_rMem test, which cares about memory only;otherwise we could fail due to other problems (e.g. not enough disk).
Add two more unittests for Node add/remove
These just test that add/remove are idempotent. This is not perfect,as we use unsorted lists for some values (instead of sets), so whenusing non-empty nodes this would break (but for empty nodes, a1-element list is sorted, so it's fine)....
Shorten some function names
Since we use the functions always module-qualified, let's drop the`instance` prefix and use a shorter one.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
htools: add spindle-related attribute to nodes
This adds the spindleCount, hiSpindles and instSpindles attributes.
The spindleCount is equivalent to spindle_count on the ganeti side (anode parameter). hiSpindles is the maximum instance-used spindles, and...
Add spindle ratio to htool's IPolicy
Just the usual data/type declarations, read/save in the text backend,etc. Also does a bit of unit-test cleanup (the way we buildipolicies).
Fix integer overflow in Node unittest
When using a really random "extra cpus" variable, this can overflow(especially on 32 bit) and thus the final value can be < 0, thus thetest will fail to fail.
It doesn't make sense to test with huge values, so let's just generate...
Add vcpu_ratio definition to the IPolicy type
This will be "naturally" read in the JSON-based backends (Luxi/Rapi),and we add explicit code for handling it in the Text backend.
Remove Node.mCpu, replaced with the ipol vcpu_ratio
Since the iPolicy contains the vcpu_ratio now, we can remove theper-node mCpu value (which was the initial way to set this attribute).
Furthermore, we need to re-set the hiCpu value when changing thepolicy....
Expand the tiered alloc unittest to check allocation stats
This was missing before, so with the new metric (NCpus), it's a goodtime to test this.
Another small optimisation in test data generation
It seems that QuickCheck will happily generate huge data structures,if that's what one requests. For this test, where we ask for [[Char]],it could happen that we get a huge number of chars, so let's limit...
Generate tags using the helper in Node tests
Otherwise we test very big lists (slow) without any advantage in doingso.
Combine two Node unittests
These two tests are very similar, so let's run them together insteadof generating the data twice. Also improve the test a bit.
Add unit-tests for --help/--version as common options
After reorganizing the way we process options, we can finally add atest that all program personalities accept --help and --version andthat they exit early for these two.
This trivial patch has a higher than expected impact on the coverage,...
Add some very basic options test
Just tests that passing a string argument returns it correctly. We'renot interested in the flags themselves that much just that thefunction itself parses some options at least correctly.
Add test for CLI.parseYesNo
Also fix a tiny typo in the docstring.
Fix bug in CLI.parseISpecString and add test
'zip' finishes on the shortest list, so parseISpecString accepted anystring of 3 values or more, which is not good. Fix this and add testfor both successful and failed parsing.
Add a unittest for the 'Simu' backend
This only test successful parsing, but for this case it testsreasonably well the resulting cluster properties, I think (coverageis >70% for expressions).
Further optimise instance test data generation
Following on the lessons from this week (commit 7806125e), this patchreplaces generation of instances `suchThat` they are smaller than onnode with direct generation of instances with upper bounds onmemory/disk/cpu....
Fix very slow unit-test data generation in some cases
Learning, learning. It turns out that for objects with even just fivefields, generating them and filtering can be costly, especially whenthe filtering is nested (as it was in this case, arbitrary1 →...
Fix one bad unit-test data generation case
It could happen that the node is generated with 0 disk available, inwhich case the operation will fail due to disk issues, not CPU. Fixthis by using the recently-introduced genOnlineNode helper.
Further JSON unit-tests for various Types.hs types
Add a unit-test helper function
… instead of always using "printTestCase … False".
Extend node evacuation unittests
Currently they only check for change secondary, via iallocator'srelocate method. We add two separate unittests that checks all thenode evacuate modes (ChangeAll, ChangePrimary) and also the groupchange request. All of these are done only for DRBD instances....
Fix a few issues in unittest data generation
This patch fixes two ugly but non-critical issues:
- the getFQDN wrongly generated only the first component as limited-length name, the rest could be any length, and that makes the generated name bad for display purposes...
Fix another data generation issue in tests
Another "good" seed value manages to generate nodes with identicalnames, which of course makes the sets be of smaller value than thelists. Avoid this issue by generating unique node names.
Add a help function for generating test nodes
This abstracts the concept of "give an sane, online node" in a singleplace, rather than spread around the tests. We also change thesignature for isNodeBig for easier usage.
Generate clusters with unique node names in tests
Nothing breaks otherwise in the cluster routines, but serialisationto/from text gets confused otherwise.
Add a complex allocation/serialisation/load test
This is unfortunately a complex test that checks if:
- starting from a empty cluster- allocating a few instances on it- serialising it via the Text backend- loading it back into internal data structures...
Implement saving/restoring of policies in Text backend
This adds a new optional (so we're backwards-compat) section to thecluster data so that we can reliably save/restore the policy data.
Note that different from the other sections, we save all policies...
Further unittests for ipolicy/ispec text serialisation
Unittests for the instance policy checks on allocation
Set group policy on nodes
In order to be able to use the group policy during allocation, we havea few choices:
- pass the group policy as a separate argument to all involved functions, or- add the policy as an attribute to nodes, or- add the group (not only the group index) as an attribute to nodes...
Add ipolicy attribute to group objects
This is straightforward; as before, so far the text and simu backendshave no way to use a non-default ipolicy.
Furthermore, we do some unit-test preparation for future changes(these seem a no-op for now, but they are needed for later patches)....
Extend ClusterData with the cluster instance policy
This attribute is always initialised to the default, and is not (yet)read/saved in the various backends.
Stop exporting JSON functionality from Utils.hs
This completes the Utils/JSON split started in commit f047f90f. Theimport graph should be cleaner now.
More reshuffling of code
Following the split Types/BasicTypes, we can remove the lastJSON-related stuff from Utils.hs, and do some more cleanup.
Some small test simplifications
Cleanup hlint errors
First, we update the recommended hlint version to what I used to get aclean output (1.8.15). Most of the changes are:
- remove unneeded parentheses- some simplifications (intercalate " " → unwords, maybe … id → fromMaybe, etc.)- removal of some duplicate code (in previous patches)...
Fix unittest bug related to offline instances
Currently, the code in Node.hs is overly strict: once a node's freememory reaches 0, it will refuse to add any instances (offline ornot). I think this is a safe safeguard (I don't expect nodes to runwithout at least 1MB of free memory), so rather than change this...
htools: re-indent QC.hs
htools: Nicer arbitrary instantiation
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: change behaviour for offline instances
htools: remove 'running' from instance options
htools: change runSt type to InstanceStatus
Generalize the generation of ADT from raw types
A few minor test improvements
This patch adds a few niceties to the test suite:
- allows matching test groups case insensitive and emit warnings when we give test group names that don't match anything- add a new operator that is similar to assertEqual in Python: it...
Use TemplateHaskell to decorate tests with names
This makes error message change from "Test 4 failed …" to "Testprop_Loader_mergeData failed", which is much more readable. It alsoremoves the duplication of test suite names in the test.hs file.
Rename some helper functions for consistency
This changes the names for some helper functions so that futurepatches are touching less unrelated code. The change replacesshortened prefixes with the full type name.
Change type of Cluster.AllocSolution
Originally, this data type was used both by instance allocation (1result), and by instance relocation (many results, one perinstance). As such, the field 'asSolutions' was a list, and thevarious code paths checked whether the length of the list matches the...
htools: remove dead code
The tryEvac/evacuateInstance functions are no longer used in the newmulti-group world order, so we remove them and change the unit-test totest the actual IAllocator function.
htools: add target_node member to migrate opcode
… and failover too. Not many changes otherwise except forserialisation and unittests.
htools: add two more small unittests
This adds tests for the opToResult and eitherToResult functions fromTypes.hs, and changes two other tests for the same module to test JSONserialisation (which automatically also tests the lower-level to/fromstring conversion functions)....
htools: a few more hlint fixes
Tested only on GHC 7.x, will test on 6.1x too before commit.
htools: import the program modules in QC.hs
This adds the binaries code to the coverage, and thus the coveragefinally shows the real coverage over all logic code (except for thehtools.hs code, which is not logic code related to the algorithms, soit doesn't matter — plus it's also very small)....
htools: add optional limit to allocation count
Currently, the tieredAlloc/iterateAlloc functions will not returnuntil the allocation fails; this means unit-testing theirfunctionality (e.g. that an instance can be allocated) is slow, sincethey will allocate all possible instances....
htools: rework compCV so that we can use [Node] too
In some cases, we have a [Node.Node] list, instead of a Node.Liststructure. Instead of converting [Node.Node] to the latter, let'srework compCV and add a compCVNodes such that we skip this(potentially expensive) step....
Fix htool unittest failure due to invalid prereqs
Currently, the test prop_Node_rMem does some reserved memory checks onnodes, and for the test to work correctly (can add more instances) werequire that the node free memory is greater than zero via aprerequisite....
htools: add parseUnit function
This is similar, but not identical, to Utils.ParseUnit. The biggestdifference is that we don't round up/down; as we only use integraltypes, the result will always be rounded down.
Moreover, since (real-world) disk sizes come in SI units, the function...
htools: simplify select tests
Rather than running all possible tests in one go, split them out inthree cases: "normal", undefined default value, undefined list values(both of which require a valid response before them).
Signed-off-by: Guido Trotter <ultrotter@google.com>...
htools: infrastructure for looking up names
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: Add if' and select as functions
if' is just a functional version of the if-then-else construct. Using itwe can manipulate ifs as functions, and also build other functionality.
select allows to choose a result depending on truth value of statements,...
htools: read/save the disk template in Text backend
This requires that we change the signature of loadInst in order toproperly annotate the error messages, which in turn requires moreunittest changes.
Also, this invalidates yet again saved data files…...
htools: add DiskTemplate to instance definition
Currently always fixed to DRBD8, as that is what we assume already.
htools: add some unittests for Types.hs
htools: rename Instance.auto_balance
… to autoBalance, per the recommended style.
htools: docstring fixes and improvements
No code changes (except one definition being moved around in QC.hs)are contained in this patch.
htools: fix prop_Node_rMem corner case
This patch fixes a bug in the test specification where we allowed nodeswith zero free memory (hence no instance can be added, at all) and addsa simple labeling of the way this test can fail.
htools: fix corner case in prop_Text_Load_Instance
This unittest had a corner case where it could fail if the sameprimary/secondary node names were generated.
htools: improve more unittests
Using new functionality in QuickCheck 2 (the suchThat function), wegenerate now better test cases, such that (heh) we have no longerincomplete tests.
htools: more fixes to unsatisfiable tests
Currently the way we generate nodes in some cases is by creating atotally random node, then restricting the test based on whether the node'size' (as defined by multiples of base unit) satifies some high/lowrules. This results in hard-to-satisfy conditions, so we change this...
htools: Rework some unittests
The new scaffolding which replaced the batch driver of QuickCheck 1 nowshows how many passes we have for incomplete tests. Some tests show verylow pass counts, so we rework them to have more actually valid testcases.
htools: Switch to QuickCheck 2.x
Since current distros don't package anymore QuickCheck 1.x, let's moveto 2.x.
This requires also a few changes to the code:
- Test.QuickCheck.Batch doesn't exist anymore, so we need to write some scaffolding code to replace it...
htools: some lint fixes
Removal of duplicate parantheses, removal of extra 'do', conversion fromnested if to guards, use hierarchical imports. All per hlint.
Add instance selection list to Loader.mergeData
Implement balancing with no instance moves
Note that --no-disk-moves and --no-instance-moves are not incompatible,but if both are used no solution can possibly exist.
Replace instance states hardcoded with constants
Add some tests for the auto_balance attribute
It tests node add/remove secondary, rather than cluster-level N+1checks, but it's better than nothing.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Adeodato Simo <dato@google.com>
Read/write auto_balance via Text
This also means another change in the text format; we really shouldmove to json…
The unittests are also update for the new 9-column layout andadditionally a bit of improvement is done.
Add a new attribute to Instance.Instance
This will mirror Ganeti's be/auto_balance one, which we need to use toproperly match N+1 computations.
htools: Make opcode naming consistent with Ganeti codebase
This patch just cleans up the htools codebase to make it more consistentwith the naming of the Ganeti codebase.