Change type of program options to 'IO [Options]'
Some options have defaults that depend on the environment, and wecould handle these in two ways:
- use a place-holder value (e.g. data X a = Default | Custom a) that is later read from the environment- move the options list to IO monad, where it can read the...
Merge branch 'devel-2.6' into submit
Conflicts: Makefile.am (reordering, fixed) htools/Ganeti/Confd/Server.hs (hlint fixes on master) htools/Ganeti/Daemon.hs (hlint)...
Merge branch 'stable-2.6' into devel-2.6
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix compatibility with newer Haskell libraries
This small patch fixes compatibility with a few newer Haskell libraries:
- base 4.6, included with ghc 7.6, removed the deprecated 'catch' function from Prelude, so our "import Prelude hiding (catch)" is now...
Convert Luxi results to Ganeti errors
This a bit too complex patch converts the result of Luxi calls(submitJob, query*, etc.) from Result to ErrorResult. It thenimmediately revers this in the HTools/Backend/Luxi module, where wedon't need necessarily the full error type (just a nice error...
Move htools backends to a separate directory
Five modules under the HTools/ directories are backendimplementations, so let's move them to a separate directory, to moreclearly show the hierarchy. I wanted to do this for a while, butmerging between branches is always an issue, so let's do it know since...
Fix a few issues found by newer hlint
Testing with a newer hlint found a few minor issues; but all are real,valid recommendations:
- don't use "if cond then f x else f y", but "f (if cond then x else y)" - "if a then b else True" is equivalent to the simpler "not a || b"...
Fix setting of 'failN1' flag for corner case
This patch includes:
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
A few unittests improvements
Small simplifications of other unit tests using the (==?)operator when possible, and typo fixes.
Remove custom OpResult type/monad
Since we now have the GeneralResult as a multi-purpose monad, we canremove the custom OpResult monad, and just use 'GeneralResultFailMode' as our type. This allows removal of a few bits ofspecialised infrastructure, relying instead on the generic one....
Use exitErr instead of explicit error message and exitWith
Furthermore, a few messages have their capitalisation changed (fixed).
Signed-off-by: Helga Velroyen <helgav@google.com>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge branch 'devel-2.6' into master
Group.hs: add 'allTags'; adjust loaders and test data for it
This commit adds a Group.allTags field to store the tags of node groups,and teaches each loader backend in HTools to populate it (additionally, theIAllocator class in lib/cmdlib.py now includes tags for groups too). Test...
Instance.hs: rename 'tags' to 'exclTags', provide 'allTags'
The mergeData function in Loader.hs included a step to filter an instance'stags to include only the exclusion tags (as specified via the commandline,or cluster-level tags). Later on, code in Node.hs assumed Instance.tags to...
Merge branch 'devel-2.6'
Conflicts: NEWS: Trivial lib/tools/ensure_dirs.py: constant moved to pathutils...
Cleanup HTools.Types/BasicTypes imports
Before we reorganised the source tree, the 'Result' type was exportedfrom HTools/Types.hs. This changed during the reorg, but at that timewe didn't change the exports; instead, we kept re-exporting it fromthe old module for compatibility....
Text.hs: update field lists in parseData comments
The comments in parseData had become out of date with the implementationsof load{Group,Node,Inst}. This commit updates the field list in comments tomatch the implementations.
Signed-off-by: Dato Simó <dato@google.com>...
Cleanup network timeouts and htools imports
This patch removes the last HTools module imports from non-htools code(the HTools.Types module), but it requires an associated cleanup:using luxi-specific constants for luxi timeouts (the only effect isthat one timeout decreases from 15 to 10, the default value in the...
Split the OptComplNumeric into integer/float ones
This allows the command completion to treat them as separate optiontypes; this is not useful in build-bash-completion right now, butmight become so in the future.
Signed-off-by: Iustin Pop <iustin@google.com>...
Define the actual arguments that are supported by programs
This defines the arguments supported and then modifies the--help-completion output to include them too.
Rename Ganeti/HTools/Utils.hs to Ganeti/Utils.hs
This is, I believe, the last non-htools specific file that still livedin the htools directory; it's already widely used in non-htools code,so let's move it before we add more functionality to this module....
Add option for displaying completion information
This patch adds support for a --help-completion option, which willdisplay the defined options and their completion information, in aformat designed to be parsed easily from Python, for integration intobuild-bash-completion....
Annotate options with completion information
This is a quite boring patch, just adding annotation information toall existing options. Some of the annotations are not very good; butwe don't have support for more precise completion inbuild-bash-completion, so this is good enough....
Remove generic options from individual programs
Currently, we test and require that each individual program (hbal,etc.) defines/supports the generic options (currently --help and--version). Even with the test, this is not optimal, since it requireschanges in many places whenever we modify the list of generic options,...
Add printing of group score in hinfo
The global cluster score is less interesting than individual groupscores, for multi-group allocation purposes.
Replace a big tuple with a custom data type
This will make it easier to add new parameters.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Use autoconf-based paths from Haskell instead of constants
Future changes will change Path.hs to use an environment variable.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix bug in non-mirrored instance allocation
The function `allocateOnSingle' has a bug in the calculation of thecluster score used for deciding which of the many target nodes to usein placing the instance: it uses the original node list for the scorecalculation....
Further hlint fixes
Commit 2cdaf22, “Re-enable standard hlint warnings”, got it almostright. The only problem is that (confusingly) the default set of hintsis not in HLint.Default, but in HLint.HLint (it includes Default andsome built-ins).
After changing the lint file to correctly include the defaults, we had...
Rework CLI modules and tests
While investigating how we could test the Daemon.hs module, I realisedthat we have a very, erm, sub-optimal situation:
- HTools/CLI.hs has a nice IO/pure separation testing in cmdline parsing, which allows some basic functionality to be tested, but...
Move Version.hs up from under HTools/
This is another module that is generic, and not htools-specific.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Remove QC.hs and replace it with an auto-generated file
In order to have correct code coverage results, we must somehow importall production modules into the test runner. Until now, this was donemanually (when we didn't forget) in QC.hs.
To improve the situation, we remove QC.hs and replace it with an...
Split last two remaining tests from QC.hs
This makes QC.hs obsolete, but we will remove it in a later patch,once we automatically build a file with all the imports (to ensurecorrect coverage results).
Move JSON.hs and Compat.hs out from under HTools/
These two files are not htools-specific, so let's move them out of theHTools subdirectory/module hierarchy and directly under Ganeti.
Split most HTools test code into separate files
Except for Ganeti.HTools.JSON, which needs rename, we split all theother test suites into separate files.
We have to add another common test helper, due to import dependencies(sigh), but otherwise this split is pretty straightforward....
Split Luxi, Qlang, Ssconf and OpCodes tests
… from QC.hs into their own files, again mirroring the production codesource tree.
Split Rpc tests from QC
This required lots of other code moves, so I created it as astandalone patch.
Split one more module out of QC and add test helpers
This splits the confd/utils tests, and adds the TestCommon module forshared test code.
Split out Objects.hs from QC.hs
This is the first file split out from QC.hs - an easy one, since ithas just one test.
The patch changes the way we build hpc excludes, since now we'll havemany modules that need to be excluded, and hpc doesn't seem to be able...
Move the unittest helper to a new Test/Ganeti dir
This patch starts the move of the test haskell code from`htools/Ganeti/HTools/' to its more proper place of`htest/Test/Ganeti'.
Implement compilation of regexes at creation time
This means that the verification of the correctness of the regex isdone once, at the deserialisation/creation time, as in the Pythoncode. To do this, we have to change the FilterRegex type from an alias...
Rename the Qlang and Queryd modules
Per the new query module hierarchy, rename Qlang to Query/Language andQueryd to Query/Server. This way, all query-related functionality isnow "contained" in the Query/ directory.
Parameterize the Filter type
In preparation for introducing filtering functionality, we convert the'Filter' type from a '*' kind to a '* -> *' kind.
This allows us to define some general properties for the filter, andfor example introduce later an easy filter compilation, etc....
Putting the multiallocate pieces together
This is the final part:
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Adding allocList function
This function iterates over the tryMGAlloc function, updates the nodelist and instance list, and refeeds it recursively until no newinstances are left.
This allows us then to allocate multiple instances and see if they wouldfit together....
Add missing luxi query 'QueryFields'
This was missed; we add the definition and the de-serialisationsupport.
Improve error reporting in our JSON conversions
Reporting things such as the following in our error messages(indentation added by me, not originally present, so it's even worse):
JSArray [JSArray [JSRational False (1 % 1),JSString (JSONString {fromJSString = "a"})],...
Add a fillDict function
This is similar to the Python version, objects.py:FillDict.
Fix qualified import of Data.Map in QC.hs
As opposed to all other places in the code, we imported this asData.Map, leaving to longer names. Let's make it similar to the restof the code (import qualified Data.Map as Map).
Add two confd library tests
These test that we encode requests correctly and decode good messages,that a message with a too old/new timestamp is never accepted, andthat a message signed with a different key is never accepted.
Reduce some more code duplication and split code
The Qlang module defines ResultStatus, but it was already defined inGaneti/Luxi.hs; let's remove the duplicate definition from there sincethe proper place is in the newer module.
Also, in order to ease testing, we split some confd functions into a...
Add test for checking Haskell/Python opcode equivalence
This is a very big hack for testing the equivalence of Python andHaskell opcode definitions. See the docstring for details; I'm notvery happy with the solution but it does the job.
An alternate option would be to launch the Python code when...
Create a custom type for disk indices
While (again) trying to test Python/Haskell encoding interoperability,I found another bug: the disk index is declared in Python asht.TPositiveInt, but in Haskell just as Int, so it can take negativevalues too. Clearly we can do better, so let's add a wrapper type that...
Handle better 'null' values in optional fields
While testing Haskell⇔Python interoperability for opcodeserialisation, I found this bug: the Haskell code doesn't treatoptional fields with 'null' values as missing, which the Python codedoes, leading to differences....
Add test case for OpCode list equivalence
For now, we only test that we don't define extra opcodes in theHaskell code; once we have parity, we can enable the (for now)commented-out lines that check the reverse.
Add infrastructure for reading Python command output
This will be used in the future to make tests/assertions forPython/Haskell codebase equivalence (e.g. checking that the sameopcodes are defined, etc.).
(Side-note: this might not work in VPATH builds; I'll see how it goes)...
QCHelper: add support for defining HUnit test cases
This will allow us to use exactly the same method as for defining andusing QuickCheck properties. The differentiation is based on name,which is not very nice but is the same method used intest-framework-th, for example (so we will be able to switch...
Rename tests for nicer names in test output
With the change to test-framework, where individual tests are listed,QCHelper tries to remove a "prop_$group_" prefix from the functionnames, for shorter names. However, some test groups (LUXI) and testfunctions are not uniform, leading to long names, so we should fix...
Switch Haskell test harness to test-framework
This patch replaces our home-grown, and quite limited, test runnerinfrastructure with test-framework(http://batterseapower.github.com/test-framework/). The rationale fordoing so is as follows:
- we will need to add support for HUnit tests, so either we add more...
Simplify the Luxi client-server test
Since now we have the recvMsgExt, we can remove the handleEOFfunction.
Fix bug in the Luxi client-server test
We didn't close the server correctly, so for test sizes bigger thanthe max-open-files ulimit, the tests were failing in accept().
OpCodes: build and export a list of all opcodes
This can be used for cross-checking with the Python code forconsistency on defined opcodes.
Enable tags query over the query socket
This patch adds the tags field to the objects which were missing it(all except Cluster), implements handling the LuxiCall QueryTags, andthen enables the use of the query socket in cli.ListTags, used by allcommands, and in the RAPI client....
Change the Luxi tags kind from String to a custom type
This will allow safer code when we implement the tags query.
Remove obsolete QrViaLuxi type
The actual query definitions are now in Qlang.hs, so let's use theItemType from there instead of luxi-defined type (which is alsoincomplete).
Rename Query2.hs to Qlang.hs
While starting to use the new filter types, I realised that what iscurrently implemented is the equivalent of `lib/qlang.py', not`lib/query.py', since we only deal with data types for now and not theactual query runtime functionality (RPC, config, etc.)....
Switch Luxi Query operation to use a proper filter
Until now, since we didn't have a proper type for the encoded queryfilters, we were ignoring the filters and handled them as null values(JSNull and respectively ()). With the current Query2 implementation,...
Implement Query2 filter JSON (de)serialisation
This adds support for encoding/decoding Query2 filters to/from JSON,in (hopefully) the same format as the Python code generates.
It also adds a simple unit-test to check that this conversion isidempotent. Of note here is that, since the Filter data type is...
Re-enable standard hlint warnings
Commit 5a1e31b4 (Add infrastructure for, and two extra hlint rules)was intended to add two extra hlint rules, but I didn't realise atthat time that "--hint" when first used overrides the built-inlints. As such, since then we were basically running with just those...
Add Objects definitions for the ispec/ipolicy types
Note that since we don't have yet a way to nicely handle two-leveloptional parameters, the Filled/Partial types and filling function areall manually built.
Improve Objects.hs definitions
This adds a few missing/incomplete definitions. We're still missingthe special parameters (disk params, hvparams, os_hvp).
Improve the TH 'Container' type
This is the first part of the changes related to the 'Container' type.
We currently handle this type as follows: it's a simple type aliasover the Data.Map type, which means:
- it's easy to use the Data.Map functions to change the type...
Simple QC tests for RPC calls
Right now we're only able to test if when a node is offline, the callfails with an appropriate errror.
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Datatypes for haskell RPC calls
We introduce typeclasses for RPC call and result and create a typeclassthat binds the two together. For that we need to useMultiParamTypeClasses and FunctionalDependencies language pragmas, whichallow us to ensure that RPC result type can be deduced based on the...
Add a server-side Luxi implementation
This is a trivial code change, but it allows us to finally test thesend-receive code on both client and server sides via a simplein-process server.
The unittest works, but it won't handle timeouts very nicely; it will...
Switch job IDs to numeric
This has been a long-standing cleanup item, which we've alwaysrefrained from doing due to the high estimated effort needed.
In reality, it turned out that after some infrastructure improvements(the previous patches), the actual job queue-related changes are quite...
Introduce a type for the ganeti job type
This will be used for easier change later.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix 'explicitely' common typo
It seems that 'explicitely' is wrong, and that the right form is'explicitly'. This is just fixing the typo plus adjusting affectedparagraphs.
Merge branch 'stable-2.6'
Reorganise the lookup functions
Currently, the LookupResult, MatchPriority and related functions arelocate in Loader.hs, since (so far) only hbal needs them in theselection of instances. However, with the new functionality on confdside, we need these functions there too, but we don't want to import...
Replace a few explicit case expressions
Since we're just talking about converting Maybe into another monad, wecan do that via the maybe function, instead of explicit casing.
Fix Haskell coverage results
There are two current issues with the coverage values:
- we don't import all modules, thus leading to incomplete coverage results (too optimistic);- we use hpc in its default mode (intersection), which means that even modules which do have coverage results but are not used in all...
Cleanup the QC.hs file w.r.t. compiler options
Currently, we build the Haskell unittests with custom GHC flags,because we (I) were quite lazy when initially writing theunittests. This is not a problem for the tests themselves, but itcreates problem when (for example) one would want to pass all...
hbal: return exit status 0 in case of early exit
This derives from an internal bug, but the story is consistent acrossboth internal and external usage of hbal.
Basically right now, hbal returns exit code 1 if requested to exitearly, even if all jobs are successful. This is counter-intuitive due...
hcheck: add two simple type aliases for readability
The same types are reused a couple of times, so let's add a couple oftype aliases for easier change later and readability.
hcheck: rework output mode
Looking at the output of hcheck, in human readable mode, it looks likeit always starts with a blank line. This is not nice, so I wanted toredo this to start cleanly.
However, looking at the code, I realised that we need some internal...
hcheck: reword and fix typo in instance/offline msg
Add hlint warning for wrong use of 'length'
While removing split instances check from hcheck, I saw a wrong use of'length': this should only be used when one needs the actual length ofthe list, and not when one wants to check whether the list is empty or...
Remove can_rebalance functionality from hcheck
Since hbal can now run rebalances even in the presence of splitinstances, we can remove this check and always run the rebalance.
Correct a wrong --help option for --simulate
This was not update when the alloc policy was added to the parameter.
hbal: relax restrictions on split instances
The recent patch series on hcheck made me realise how much pain wehave due to hbal not being able to work (at all) with split instances.
This is sub-optimal; ideally, hbal would degrade its behaviour, butnot completely refuse to work. This patch does this, by marking any...
Fix a few style issue in hcheck
This fixes a couple of issue I've seen while fixing the List import:
- removes over-use of printf with putStr/show- wraps some lines- removes superfluous parentheses in constructs like 'IO (a)'- makes sure that in type signatures, the '->' are placed at the...
Fix old-style import
Commit 1213f9d6 (re)added an old-style import, which fails with recentcompilers.
Style fix in hcheck
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Simplify stats printing
PrintGroupStats and printClusterStats had very similar codestructure. This patch abstracts those two into printStats, making ita little more generic. Also, printStats has its arguments documentedas requested in previous patch series....
Cleaner simulation of rebalance
This replaces the 'if .. then .. else ..' with a large block of code in'else' branch by separating this code into a separate function.
Small reordering and renaming in hcheck
Reordered functions and shortened one function name.