Implement compilation of regexes at creation time
This means that the verification of the correctness of the regex isdone once, at the deserialisation/creation time, as in the Pythoncode. To do this, we have to change the FilterRegex type from an alias...
Rename the Qlang and Queryd modules
Per the new query module hierarchy, rename Qlang to Query/Language andQueryd to Query/Server. This way, all query-related functionality isnow "contained" in the Query/ directory.
Signed-off-by: Iustin Pop <iustin@google.com>...
Parameterize the Filter type
In preparation for introducing filtering functionality, we convert the'Filter' type from a '*' kind to a '* -> *' kind.
This allows us to define some general properties for the filter, andfor example introduce later an easy filter compilation, etc....
Putting the multiallocate pieces together
This is the final part:
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Adding allocList function
This function iterates over the tryMGAlloc function, updates the nodelist and instance list, and refeeds it recursively until no newinstances are left.
This allows us then to allocate multiple instances and see if they wouldfit together....
Add missing luxi query 'QueryFields'
This was missed; we add the definition and the de-serialisationsupport.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Improve error reporting in our JSON conversions
Reporting things such as the following in our error messages(indentation added by me, not originally present, so it's even worse):
JSArray [JSArray [JSRational False (1 % 1),JSString (JSONString {fromJSString = "a"})],...
Add a fillDict function
This is similar to the Python version, objects.py:FillDict.
Fix qualified import of Data.Map in QC.hs
As opposed to all other places in the code, we imported this asData.Map, leaving to longer names. Let's make it similar to the restof the code (import qualified Data.Map as Map).
Add two confd library tests
These test that we encode requests correctly and decode good messages,that a message with a too old/new timestamp is never accepted, andthat a message signed with a different key is never accepted.
Reduce some more code duplication and split code
The Qlang module defines ResultStatus, but it was already defined inGaneti/Luxi.hs; let's remove the duplicate definition from there sincethe proper place is in the newer module.
Also, in order to ease testing, we split some confd functions into a...
Add test for checking Haskell/Python opcode equivalence
This is a very big hack for testing the equivalence of Python andHaskell opcode definitions. See the docstring for details; I'm notvery happy with the solution but it does the job.
An alternate option would be to launch the Python code when...
Create a custom type for disk indices
While (again) trying to test Python/Haskell encoding interoperability,I found another bug: the disk index is declared in Python asht.TPositiveInt, but in Haskell just as Int, so it can take negativevalues too. Clearly we can do better, so let's add a wrapper type that...
Handle better 'null' values in optional fields
While testing Haskell⇔Python interoperability for opcodeserialisation, I found this bug: the Haskell code doesn't treatoptional fields with 'null' values as missing, which the Python codedoes, leading to differences....
Add test case for OpCode list equivalence
For now, we only test that we don't define extra opcodes in theHaskell code; once we have parity, we can enable the (for now)commented-out lines that check the reverse.
Add infrastructure for reading Python command output
This will be used in the future to make tests/assertions forPython/Haskell codebase equivalence (e.g. checking that the sameopcodes are defined, etc.).
(Side-note: this might not work in VPATH builds; I'll see how it goes)...
QCHelper: add support for defining HUnit test cases
This will allow us to use exactly the same method as for defining andusing QuickCheck properties. The differentiation is based on name,which is not very nice but is the same method used intest-framework-th, for example (so we will be able to switch...
Rename tests for nicer names in test output
With the change to test-framework, where individual tests are listed,QCHelper tries to remove a "prop_$group_" prefix from the functionnames, for shorter names. However, some test groups (LUXI) and testfunctions are not uniform, leading to long names, so we should fix...
Switch Haskell test harness to test-framework
This patch replaces our home-grown, and quite limited, test runnerinfrastructure with test-framework(http://batterseapower.github.com/test-framework/). The rationale fordoing so is as follows:
- we will need to add support for HUnit tests, so either we add more...
Simplify the Luxi client-server test
Since now we have the recvMsgExt, we can remove the handleEOFfunction.
Fix bug in the Luxi client-server test
We didn't close the server correctly, so for test sizes bigger thanthe max-open-files ulimit, the tests were failing in accept().
OpCodes: build and export a list of all opcodes
This can be used for cross-checking with the Python code forconsistency on defined opcodes.
Enable tags query over the query socket
This patch adds the tags field to the objects which were missing it(all except Cluster), implements handling the LuxiCall QueryTags, andthen enables the use of the query socket in cli.ListTags, used by allcommands, and in the RAPI client....
Change the Luxi tags kind from String to a custom type
This will allow safer code when we implement the tags query.
Remove obsolete QrViaLuxi type
The actual query definitions are now in Qlang.hs, so let's use theItemType from there instead of luxi-defined type (which is alsoincomplete).
Rename Query2.hs to Qlang.hs
While starting to use the new filter types, I realised that what iscurrently implemented is the equivalent of `lib/qlang.py', not`lib/query.py', since we only deal with data types for now and not theactual query runtime functionality (RPC, config, etc.)....
Switch Luxi Query operation to use a proper filter
Until now, since we didn't have a proper type for the encoded queryfilters, we were ignoring the filters and handled them as null values(JSNull and respectively ()). With the current Query2 implementation,...
Implement Query2 filter JSON (de)serialisation
This adds support for encoding/decoding Query2 filters to/from JSON,in (hopefully) the same format as the Python code generates.
It also adds a simple unit-test to check that this conversion isidempotent. Of note here is that, since the Filter data type is...
Re-enable standard hlint warnings
Commit 5a1e31b4 (Add infrastructure for, and two extra hlint rules)was intended to add two extra hlint rules, but I didn't realise atthat time that "--hint" when first used overrides the built-inlints. As such, since then we were basically running with just those...
Add Objects definitions for the ispec/ipolicy types
Note that since we don't have yet a way to nicely handle two-leveloptional parameters, the Filled/Partial types and filling function areall manually built.
Improve Objects.hs definitions
This adds a few missing/incomplete definitions. We're still missingthe special parameters (disk params, hvparams, os_hvp).
Improve the TH 'Container' type
This is the first part of the changes related to the 'Container' type.
We currently handle this type as follows: it's a simple type aliasover the Data.Map type, which means:
- it's easy to use the Data.Map functions to change the type...
Simple QC tests for RPC calls
Right now we're only able to test if when a node is offline, the callfails with an appropriate errror.
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Datatypes for haskell RPC calls
We introduce typeclasses for RPC call and result and create a typeclassthat binds the two together. For that we need to useMultiParamTypeClasses and FunctionalDependencies language pragmas, whichallow us to ensure that RPC result type can be deduced based on the...
Merge branch 'devel-2.6'
Add a server-side Luxi implementation
This is a trivial code change, but it allows us to finally test thesend-receive code on both client and server sides via a simplein-process server.
The unittest works, but it won't handle timeouts very nicely; it will...
Switch job IDs to numeric
This has been a long-standing cleanup item, which we've alwaysrefrained from doing due to the high estimated effort needed.
In reality, it turned out that after some infrastructure improvements(the previous patches), the actual job queue-related changes are quite...
Introduce a type for the ganeti job type
This will be used for easier change later.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix 'explicitely' common typo
It seems that 'explicitely' is wrong, and that the right form is'explicitly'. This is just fixing the typo plus adjusting affectedparagraphs.
Merge branch 'stable-2.6'
Reorganise the lookup functions
Currently, the LookupResult, MatchPriority and related functions arelocate in Loader.hs, since (so far) only hbal needs them in theselection of instances. However, with the new functionality on confdside, we need these functions there too, but we don't want to import...
Replace a few explicit case expressions
Since we're just talking about converting Maybe into another monad, wecan do that via the maybe function, instead of explicit casing.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Fix Haskell coverage results
There are two current issues with the coverage values:
- we don't import all modules, thus leading to incomplete coverage results (too optimistic);- we use hpc in its default mode (intersection), which means that even modules which do have coverage results but are not used in all...
Cleanup the QC.hs file w.r.t. compiler options
Currently, we build the Haskell unittests with custom GHC flags,because we (I) were quite lazy when initially writing theunittests. This is not a problem for the tests themselves, but itcreates problem when (for example) one would want to pass all...
hbal: return exit status 0 in case of early exit
This derives from an internal bug, but the story is consistent acrossboth internal and external usage of hbal.
Basically right now, hbal returns exit code 1 if requested to exitearly, even if all jobs are successful. This is counter-intuitive due...
hcheck: add two simple type aliases for readability
The same types are reused a couple of times, so let's add a couple oftype aliases for easier change later and readability.
hcheck: rework output mode
Looking at the output of hcheck, in human readable mode, it looks likeit always starts with a blank line. This is not nice, so I wanted toredo this to start cleanly.
However, looking at the code, I realised that we need some internal...
hcheck: reword and fix typo in instance/offline msg
Add hlint warning for wrong use of 'length'
While removing split instances check from hcheck, I saw a wrong use of'length': this should only be used when one needs the actual length ofthe list, and not when one wants to check whether the list is empty or...
Remove can_rebalance functionality from hcheck
Since hbal can now run rebalances even in the presence of splitinstances, we can remove this check and always run the rebalance.
Correct a wrong --help option for --simulate
This was not update when the alloc policy was added to the parameter.
hbal: relax restrictions on split instances
The recent patch series on hcheck made me realise how much pain wehave due to hbal not being able to work (at all) with split instances.
This is sub-optimal; ideally, hbal would degrade its behaviour, butnot completely refuse to work. This patch does this, by marking any...
Fix a few style issue in hcheck
This fixes a couple of issue I've seen while fixing the List import:
- removes over-use of printf with putStr/show- wraps some lines- removes superfluous parentheses in constructs like 'IO (a)'- makes sure that in type signatures, the '->' are placed at the...
Fix old-style import
Commit 1213f9d6 (re)added an old-style import, which fails with recentcompilers.
Style fix in hcheck
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Simplify stats printing
PrintGroupStats and printClusterStats had very similar codestructure. This patch abstracts those two into printStats, making ita little more generic. Also, printStats has its arguments documentedas requested in previous patch series....
Cleaner simulation of rebalance
This replaces the 'if .. then .. else ..' with a large block of code in'else' branch by separating this code into a separate function.
Small reordering and renaming in hcheck
Reordered functions and shortened one function name.
Move 'need rebalance' check from cluster check
As suggested by Rene, we should keep the check separate from printing.
Always print all variables in machine readable version
As discussed in cover letter of patch series introducing hcheck, inmachine readable version we should always print values of all variables.When hbal simulation was not run, FINAL values are equal to INIT values....
Print idx <-> uuid mappings in machine readable mode
As per Iustin Pop's suggestion, we are printing group mappingsfrom group idx (known only to htools) to group uuid, which identifiesthe group uniquely. This is required because group uuid and namemay contain characters such as '-' which are not valid parts of...
Human readable group name improved
Instead of printing group id in human readable version of hcheck,we now print group name.
Print info about ability to rebalance
As suggested during offline discussion, this patch adds a new bashvariable and description field in hcheck, which informs if therebalance was possible. This is to determine if there are splitinstances on the cluster, in which case rebalance is impossible....
Improve readability for hcheck-related changes
As suggested by Rene, I added comments to types for printKeysand printFinal. Also, realigned some lines in Hspace.
Machine readable hcheck functionality
Machine readable output for hcheck.
Human readable hcheck functionality
Full hcheck functionality is implemented in this patch. Hcheck runsper-group checks, provides summary for the entire cluster and simulatesrebalance if required, the printing the modified metrics.In this patch, only the human-readable output is available....
Allowing rebalance to run silently
Part of hcheck's simulation required us to silently run hbal. For thatpurpose we expose iterateDepth and allow it to run silently.
Generalize functions used by both hspace and hcheck
As hspace and hcheck both use machine readable options, they requiresimilar functions - which are now generalized and moved to CLI andUtils.
Initial commit for introducting hcheck tool
Introduce infrastructure required to add Hcheck and build itsuccessfuly.
Fix corner case in quick-check generated tests
Thanks to Agata Murawska, a random seed that generates bad test caseswas found ("seed was 1768143307 2118231514, test size 74"). This isdue to the fact that some nodes were generated with tMem=0, fMem=0,which means that pMem was NaN and thus the node was not comparing...
Fix prefix bug in Haskell ssconf implementation
This patch fixes a (brown-bag) bug in the ssconf implementation wherethe looked-for filenames were not prefixed with the ssconf fileprefix.
Additionally, a test for this is added (which fails without the fix)....
Add decoding of Luxi calls and unittests for LuxiOp
This patch adds a hand-coded decoder for LuxiCall arguments, as thedata-structure is not uniform enough for automated generation (evenfor the serialisation, we had to add hints for some fields,de-serialisation is even harder)....
Some unit tests improvements
In preparation for unit-testing the Luxi definitions, this improvesthe auto-generation of OpCodes: currently this uses unbounded plainStrings for names and fields, and this creates too big test sizes(e.g. when automatically generating job sets)....
Add a new JSON function
And its associated unittests.
Switch to new-style exception handling
Currently, we're using Prelude.catch to handle I/O errors inhtools. This style of error handling has been deprecated for a while,but it still used to work without warnings.
However, the GHC release 7.4 has started to emit deprecation warnings...
Rework exit model
While updating the confd code, I realised that we have lots ofduplication in the exit model for the various programs.
So this patch attempts to abstract all the exits via a couple of newfunctions; sorry for the somewhat big patch, but I hope the payoff is...
Allow hail to read data from stdin
This patch makes hail treat '-' as denoting stdin, per the usual Unixconvention. This will help with testing.
hspace: Make use of the spindle_use
htools: Fix some compiler warnings
Enhance the RAPI backend to accept file:// URLs
This will allow offline testing of this backend (except for the actualcurl part), also in the case where we didn't compile against curl.
htools: Reflect the rename of spindle_usage to spindle_use
Add a helper function for mirrorType computations
We most always use `templateMirrorType . diskTemplate`, so let's add ahelper function for this.
Enable all disk templates in unit-tests
After the many patches implementing the shared storage functionalityin iallocator, balances, etc., and after the patches preparing theunit-tests to do the right thing for shared storage, we can finallyflip the bits that make instances have "random" disk templates as...
Enable evacuation/relocation for shared storage
This patch toggles the final bit and enables nodeEvacInstance to workon shared storage instances. Diskless instances are handled the same(as in, None is the same everywhere :).
Replace explicit case with 'either'
Instead of manually case-ing on the Either contents, let's just useeither with const functions (not really readable, but…).
Rename evacDrbdSecondaryInner to evacOneNodeInner
The function is now generic, so let's rename it and cleanup/improvethe docstrings.
Abstract the secondary evac into a separate function
This is generic enough (no longer depending on the replace type) thatit will be used for other replace modes too. So all we need to do isto abstract it to a separate function (that can be called from...
Fix Node_addPriFD test for arbitrary disk templates
Currently, this test assumes localStorageTemplates, so let's make surewe enforce that on the given instance
Extend the possible move types with FailoverToAny
This brings together all previous pieces and allows balancing to work(with the known caveats related to disk space calculation) for sharedstorage disk templates.
htools: Add the new ipolicy field
Also add the spindle usage to QC
htools: Use the spindle usage in the algorithms
In the same transition also do a simple refactor of the caculation
htools: Convert all the other backends as this is trivial
htools: Adapt the Text backend to spindle usage
The old text files remain compatible with this change.
htools: Adding spindle usage to Instance
For now we just pass in "1" as spindle usage and adapt the backendlater.
Rework unit parsing
Due to how conversions were implemented previously, 1TB failed toparse on 32-bit, as we were overflowing during computation, eventhough the final result would fit easily.
This patch moves the parsing of the scaling factor to a separate...
htools: Make it possible to load an IAlloc request
This is only useful in the stand-alone command line tools:
- hbal- hspace- hinfo
This change also documents the flag in all man-pages.
Fix relocation test ClusterAllocRelocate
As for the evacuate tests, we require a mirrored disk template, butotherwise the test should work for both mirror types.
Additionally, we perform a simplification that was left as TODO.
Fix IAlloc.processRelocate to support shared storage
Currently, this function expects (but doesn't really check! only viasecondary node being defined) a DRBD8 template type.
This patch (not very readable, sorry) changes it so that we checkwhether we're internally mirrored, externally mirrored, or not...
Fix up Cluster nodeEvac unittests for shared storage
Currently, these presume that the instance being generated always isDRBD/relocatable (but won't be in the future), and also presume thatall relocation modes can be used (hint: not ☺).
So we add a couple of helper functions to restrict the used disk...