History | View | Annotate | Download (84.9 kB)
Parameterize the Filter type
In preparation for introducing filtering functionality, we convert the'Filter' type from a '*' kind to a '* -> *' kind.
This allows us to define some general properties for the filter, andfor example introduce later an easy filter compilation, etc....
Add missing luxi query 'QueryFields'
This was missed; we add the definition and the de-serialisationsupport.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Add a fillDict function
This is similar to the Python version, objects.py:FillDict.
Fix qualified import of Data.Map in QC.hs
As opposed to all other places in the code, we imported this asData.Map, leaving to longer names. Let's make it similar to the restof the code (import qualified Data.Map as Map).
Signed-off-by: Iustin Pop <iustin@google.com>...
Add two confd library tests
These test that we encode requests correctly and decode good messages,that a message with a too old/new timestamp is never accepted, andthat a message signed with a different key is never accepted.
Reduce some more code duplication and split code
The Qlang module defines ResultStatus, but it was already defined inGaneti/Luxi.hs; let's remove the duplicate definition from there sincethe proper place is in the newer module.
Also, in order to ease testing, we split some confd functions into a...
Add test for checking Haskell/Python opcode equivalence
This is a very big hack for testing the equivalence of Python andHaskell opcode definitions. See the docstring for details; I'm notvery happy with the solution but it does the job.
An alternate option would be to launch the Python code when...
Create a custom type for disk indices
While (again) trying to test Python/Haskell encoding interoperability,I found another bug: the disk index is declared in Python asht.TPositiveInt, but in Haskell just as Int, so it can take negativevalues too. Clearly we can do better, so let's add a wrapper type that...
Add test case for OpCode list equivalence
For now, we only test that we don't define extra opcodes in theHaskell code; once we have parity, we can enable the (for now)commented-out lines that check the reverse.
Add infrastructure for reading Python command output
This will be used in the future to make tests/assertions forPython/Haskell codebase equivalence (e.g. checking that the sameopcodes are defined, etc.).
(Side-note: this might not work in VPATH builds; I'll see how it goes)...
Rename tests for nicer names in test output
With the change to test-framework, where individual tests are listed,QCHelper tries to remove a "prop_$group_" prefix from the functionnames, for shorter names. However, some test groups (LUXI) and testfunctions are not uniform, leading to long names, so we should fix...
Simplify the Luxi client-server test
Since now we have the recvMsgExt, we can remove the handleEOFfunction.
Fix bug in the Luxi client-server test
We didn't close the server correctly, so for test sizes bigger thanthe max-open-files ulimit, the tests were failing in accept().
OpCodes: build and export a list of all opcodes
This can be used for cross-checking with the Python code forconsistency on defined opcodes.
Enable tags query over the query socket
This patch adds the tags field to the objects which were missing it(all except Cluster), implements handling the LuxiCall QueryTags, andthen enables the use of the query socket in cli.ListTags, used by allcommands, and in the RAPI client....
Change the Luxi tags kind from String to a custom type
This will allow safer code when we implement the tags query.
Remove obsolete QrViaLuxi type
The actual query definitions are now in Qlang.hs, so let's use theItemType from there instead of luxi-defined type (which is alsoincomplete).
Rename Query2.hs to Qlang.hs
While starting to use the new filter types, I realised that what iscurrently implemented is the equivalent of `lib/qlang.py', not`lib/query.py', since we only deal with data types for now and not theactual query runtime functionality (RPC, config, etc.)....
Switch Luxi Query operation to use a proper filter
Until now, since we didn't have a proper type for the encoded queryfilters, we were ignoring the filters and handled them as null values(JSNull and respectively ()). With the current Query2 implementation,...
Implement Query2 filter JSON (de)serialisation
This adds support for encoding/decoding Query2 filters to/from JSON,in (hopefully) the same format as the Python code generates.
It also adds a simple unit-test to check that this conversion isidempotent. Of note here is that, since the Filter data type is...
Re-enable standard hlint warnings
Commit 5a1e31b4 (Add infrastructure for, and two extra hlint rules)was intended to add two extra hlint rules, but I didn't realise atthat time that "--hint" when first used overrides the built-inlints. As such, since then we were basically running with just those...
Add Objects definitions for the ispec/ipolicy types
Note that since we don't have yet a way to nicely handle two-leveloptional parameters, the Filled/Partial types and filling function areall manually built.
Improve Objects.hs definitions
This adds a few missing/incomplete definitions. We're still missingthe special parameters (disk params, hvparams, os_hvp).
Simple QC tests for RPC calls
Right now we're only able to test if when a node is offline, the callfails with an appropriate errror.
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Datatypes for haskell RPC calls
We introduce typeclasses for RPC call and result and create a typeclassthat binds the two together. For that we need to useMultiParamTypeClasses and FunctionalDependencies language pragmas, whichallow us to ensure that RPC result type can be deduced based on the...
Add a server-side Luxi implementation
This is a trivial code change, but it allows us to finally test thesend-receive code on both client and server sides via a simplein-process server.
The unittest works, but it won't handle timeouts very nicely; it will...
Merge branch 'stable-2.6'
Reorganise the lookup functions
Currently, the LookupResult, MatchPriority and related functions arelocate in Loader.hs, since (so far) only hbal needs them in theselection of instances. However, with the new functionality on confdside, we need these functions there too, but we don't want to import...
Fix Haskell coverage results
There are two current issues with the coverage values:
- we don't import all modules, thus leading to incomplete coverage results (too optimistic);- we use hpc in its default mode (intersection), which means that even modules which do have coverage results but are not used in all...
Cleanup the QC.hs file w.r.t. compiler options
Currently, we build the Haskell unittests with custom GHC flags,because we (I) were quite lazy when initially writing theunittests. This is not a problem for the tests themselves, but itcreates problem when (for example) one would want to pass all...
Fix corner case in quick-check generated tests
Thanks to Agata Murawska, a random seed that generates bad test caseswas found ("seed was 1768143307 2118231514, test size 74"). This isdue to the fact that some nodes were generated with tMem=0, fMem=0,which means that pMem was NaN and thus the node was not comparing...
Fix prefix bug in Haskell ssconf implementation
This patch fixes a (brown-bag) bug in the ssconf implementation wherethe looked-for filenames were not prefixed with the ssconf fileprefix.
Additionally, a test for this is added (which fails without the fix)....
Add decoding of Luxi calls and unittests for LuxiOp
This patch adds a hand-coded decoder for LuxiCall arguments, as thedata-structure is not uniform enough for automated generation (evenfor the serialisation, we had to add hints for some fields,de-serialisation is even harder)....
Some unit tests improvements
In preparation for unit-testing the Luxi definitions, this improvesthe auto-generation of OpCodes: currently this uses unbounded plainStrings for names and fields, and this creates too big test sizes(e.g. when automatically generating job sets)....
Add a new JSON function
And its associated unittests.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
htools: Reflect the rename of spindle_usage to spindle_use
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add a helper function for mirrorType computations
We most always use `templateMirrorType . diskTemplate`, so let's add ahelper function for this.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Enable all disk templates in unit-tests
After the many patches implementing the shared storage functionalityin iallocator, balances, etc., and after the patches preparing theunit-tests to do the right thing for shared storage, we can finallyflip the bits that make instances have "random" disk templates as...
Fix Node_addPriFD test for arbitrary disk templates
Currently, this test assumes localStorageTemplates, so let's make surewe enforce that on the given instance
htools: Add the new ipolicy field
Also add the spindle usage to QC
htools: Adapt the Text backend to spindle usage
The old text files remain compatible with this change.
htools: Adding spindle usage to Instance
For now we just pass in "1" as spindle usage and adapt the backendlater.
Rework unit parsing
Due to how conversions were implemented previously, 1TB failed toparse on 32-bit, as we were overflowing during computation, eventhough the final result would fit easily.
This patch moves the parsing of the scaling factor to a separate...
Fix relocation test ClusterAllocRelocate
As for the evacuate tests, we require a mirrored disk template, butotherwise the test should work for both mirror types.
Additionally, we perform a simplification that was left as TODO.
Fix up Cluster nodeEvac unittests for shared storage
Currently, these presume that the instance being generated always isDRBD/relocatable (but won't be in the future), and also presume thatall relocation modes can be used (hint: not ☺).
So we add a couple of helper functions to restrict the used disk...
Fix a few tests that hard-coded the number of nodes
Instead of randomly generating the number of required nodes orhard-coding it, compute it from the disk template of theinstance. Otherwise tests will fail when the disk template is notDRBD8 and we ask for allocation on 2 nodes....
Fix test Node_addSec_idempotent
Another test that only works for DRBD8 instances.
Fix the Node_rMem test to specify DRBD8 templates
… since memory reservation only is used for this template.
Split the Node_addOffline test into pri/sec
Currently, this (single) test checks whether we can add an instance asprimary and secondary to a given node. This won't work for instancetemplates other than DRBD8, so we split it into:
- an add to primary test, which should work for all disk templates...
Do not rely on no-secondary for movable tests
Currently, the code in Loader.hs marks as un-movable any instanceswhich don't have a secondary. This is not valid for any shared storagetemplates, so let's remove that check completely. The initial state of...
Fixup a Node unittest
This fixes the prop_Node_rMem test, which cares about memory only;otherwise we could fail due to other problems (e.g. not enough disk).
Add two more unittests for Node add/remove
These just test that add/remove are idempotent. This is not perfect,as we use unsorted lists for some values (instead of sets), so whenusing non-empty nodes this would break (but for empty nodes, a1-element list is sorted, so it's fine)....
Shorten some function names
Since we use the functions always module-qualified, let's drop the`instance` prefix and use a shorter one.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
htools: add spindle-related attribute to nodes
This adds the spindleCount, hiSpindles and instSpindles attributes.
The spindleCount is equivalent to spindle_count on the ganeti side (anode parameter). hiSpindles is the maximum instance-used spindles, and...
Add spindle ratio to htool's IPolicy
Just the usual data/type declarations, read/save in the text backend,etc. Also does a bit of unit-test cleanup (the way we buildipolicies).
Fix integer overflow in Node unittest
When using a really random "extra cpus" variable, this can overflow(especially on 32 bit) and thus the final value can be < 0, thus thetest will fail to fail.
It doesn't make sense to test with huge values, so let's just generate...
Add vcpu_ratio definition to the IPolicy type
This will be "naturally" read in the JSON-based backends (Luxi/Rapi),and we add explicit code for handling it in the Text backend.
Remove Node.mCpu, replaced with the ipol vcpu_ratio
Since the iPolicy contains the vcpu_ratio now, we can remove theper-node mCpu value (which was the initial way to set this attribute).
Furthermore, we need to re-set the hiCpu value when changing thepolicy....
Expand the tiered alloc unittest to check allocation stats
This was missing before, so with the new metric (NCpus), it's a goodtime to test this.
Another small optimisation in test data generation
It seems that QuickCheck will happily generate huge data structures,if that's what one requests. For this test, where we ask for [[Char]],it could happen that we get a huge number of chars, so let's limit...
Generate tags using the helper in Node tests
Otherwise we test very big lists (slow) without any advantage in doingso.
Combine two Node unittests
These two tests are very similar, so let's run them together insteadof generating the data twice. Also improve the test a bit.
Add unit-tests for --help/--version as common options
After reorganizing the way we process options, we can finally add atest that all program personalities accept --help and --version andthat they exit early for these two.
This trivial patch has a higher than expected impact on the coverage,...
Add some very basic options test
Just tests that passing a string argument returns it correctly. We'renot interested in the flags themselves that much just that thefunction itself parses some options at least correctly.
Add test for CLI.parseYesNo
Also fix a tiny typo in the docstring.
Fix bug in CLI.parseISpecString and add test
'zip' finishes on the shortest list, so parseISpecString accepted anystring of 3 values or more, which is not good. Fix this and add testfor both successful and failed parsing.
Add a unittest for the 'Simu' backend
This only test successful parsing, but for this case it testsreasonably well the resulting cluster properties, I think (coverageis >70% for expressions).
Further optimise instance test data generation
Following on the lessons from this week (commit 7806125e), this patchreplaces generation of instances `suchThat` they are smaller than onnode with direct generation of instances with upper bounds onmemory/disk/cpu....
Fix very slow unit-test data generation in some cases
Learning, learning. It turns out that for objects with even just fivefields, generating them and filtering can be costly, especially whenthe filtering is nested (as it was in this case, arbitrary1 →...
Fix one bad unit-test data generation case
It could happen that the node is generated with 0 disk available, inwhich case the operation will fail due to disk issues, not CPU. Fixthis by using the recently-introduced genOnlineNode helper.
Further JSON unit-tests for various Types.hs types
Add a unit-test helper function
… instead of always using "printTestCase … False".
Extend node evacuation unittests
Currently they only check for change secondary, via iallocator'srelocate method. We add two separate unittests that checks all thenode evacuate modes (ChangeAll, ChangePrimary) and also the groupchange request. All of these are done only for DRBD instances....
Fix a few issues in unittest data generation
This patch fixes two ugly but non-critical issues:
- the getFQDN wrongly generated only the first component as limited-length name, the rest could be any length, and that makes the generated name bad for display purposes...
Fix another data generation issue in tests
Another "good" seed value manages to generate nodes with identicalnames, which of course makes the sets be of smaller value than thelists. Avoid this issue by generating unique node names.
Add a help function for generating test nodes
This abstracts the concept of "give an sane, online node" in a singleplace, rather than spread around the tests. We also change thesignature for isNodeBig for easier usage.
Generate clusters with unique node names in tests
Nothing breaks otherwise in the cluster routines, but serialisationto/from text gets confused otherwise.
Add a complex allocation/serialisation/load test
This is unfortunately a complex test that checks if:
- starting from a empty cluster- allocating a few instances on it- serialising it via the Text backend- loading it back into internal data structures...
Implement saving/restoring of policies in Text backend
This adds a new optional (so we're backwards-compat) section to thecluster data so that we can reliably save/restore the policy data.
Note that different from the other sections, we save all policies...
Further unittests for ipolicy/ispec text serialisation
Unittests for the instance policy checks on allocation
Set group policy on nodes
In order to be able to use the group policy during allocation, we havea few choices:
- pass the group policy as a separate argument to all involved functions, or- add the policy as an attribute to nodes, or- add the group (not only the group index) as an attribute to nodes...
Add ipolicy attribute to group objects
This is straightforward; as before, so far the text and simu backendshave no way to use a non-default ipolicy.
Furthermore, we do some unit-test preparation for future changes(these seem a no-op for now, but they are needed for later patches)....
Extend ClusterData with the cluster instance policy
This attribute is always initialised to the default, and is not (yet)read/saved in the various backends.
Stop exporting JSON functionality from Utils.hs
This completes the Utils/JSON split started in commit f047f90f. Theimport graph should be cleaner now.
More reshuffling of code
Following the split Types/BasicTypes, we can remove the lastJSON-related stuff from Utils.hs, and do some more cleanup.
Some small test simplifications
Cleanup hlint errors
First, we update the recommended hlint version to what I used to get aclean output (1.8.15). Most of the changes are:
- remove unneeded parentheses- some simplifications (intercalate " " → unwords, maybe … id → fromMaybe, etc.)- removal of some duplicate code (in previous patches)...
Fix unittest bug related to offline instances
Currently, the code in Node.hs is overly strict: once a node's freememory reaches 0, it will refuse to add any instances (offline ornot). I think this is a safe safeguard (I don't expect nodes to runwithout at least 1MB of free memory), so rather than change this...
htools: re-indent QC.hs
htools: Nicer arbitrary instantiation
htools: change behaviour for offline instances
htools: remove 'running' from instance options
htools: change runSt type to InstanceStatus
Generalize the generation of ADT from raw types
A few minor test improvements
This patch adds a few niceties to the test suite:
- allows matching test groups case insensitive and emit warnings when we give test group names that don't match anything- add a new operator that is similar to assertEqual in Python: it...
Use TemplateHaskell to decorate tests with names
This makes error message change from "Test 4 failed …" to "Testprop_Loader_mergeData failed", which is much more readable. It alsoremoves the duplication of test suite names in the test.hs file.
Rename some helper functions for consistency
This changes the names for some helper functions so that futurepatches are touching less unrelated code. The change replacesshortened prefixes with the full type name.
Change type of Cluster.AllocSolution
Originally, this data type was used both by instance allocation (1result), and by instance relocation (many results, one perinstance). As such, the field 'asSolutions' was a list, and thevarious code paths checked whether the length of the list matches the...