Simplify property and test case names
Since we now have separate namespaces due to the multi-file split, wedon't need to keep the name of the module in the property names, as wedon't have so many potential conflicts anymore.
We remove the group prefix handling from TestHelper and simply do a...
Move Version.hs up from under HTools/
This is another module that is generic, and not htools-specific.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Two tiny makefile fixes
During the recent moves and renames, two things have slipped through,since I didn't run make check-local…
Remove QC.hs and replace it with an auto-generated file
In order to have correct code coverage results, we must somehow importall production modules into the test runner. Until now, this was donemanually (when we didn't forget) in QC.hs.
To improve the situation, we remove QC.hs and replace it with an...
Split the Makefile variable HS_LIB_SRCS
This currently holds both production files and test files; since wewant to treat these separately, let's split the variable in two, andintroduce another one for the old, inclusive one.
Signed-off-by: Iustin Pop <iustin@google.com>...
Split last two remaining tests from QC.hs
This makes QC.hs obsolete, but we will remove it in a later patch,once we automatically build a file with all the imports (to ensurecorrect coverage results).
Move JSON.hs and Compat.hs out from under HTools/
These two files are not htools-specific, so let's move them out of theHTools subdirectory/module hierarchy and directly under Ganeti.
Split most HTools test code into separate files
Except for Ganeti.HTools.JSON, which needs rename, we split all theother test suites into separate files.
We have to add another common test helper, due to import dependencies(sigh), but otherwise this split is pretty straightforward....
Split Luxi, Qlang, Ssconf and OpCodes tests
… from QC.hs into their own files, again mirroring the production codesource tree.
Split Rpc tests from QC
This required lots of other code moves, so I created it as astandalone patch.
Split one more module out of QC and add test helpers
This splits the confd/utils tests, and adds the TestCommon module forshared test code.
Split out Objects.hs from QC.hs
This is the first file split out from QC.hs - an easy one, since ithas just one test.
The patch changes the way we build hpc excludes, since now we'll havemany modules that need to be excluded, and hpc doesn't seem to be able...
Move the unittest helper to a new Test/Ganeti dir
This patch starts the move of the test haskell code from`htools/Ganeti/HTools/' to its more proper place of`htest/Test/Ganeti'.
Move generated htools test stubs to htest/
This moves the last (I think) htools-related bits out of test/ underhtest/.
Move shelltests from test/ to htest/shelltests/
This is another rather trivial patch, moving all the (htools)shelltests to their own directory.
Move htools test files from test/data/htools to htest/data
Following up on the program moves, we now move the test data files.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Move haskell test code to htest/
This is the first commit of a series that will attempt to cleanup thetest code organisation, which evolved somewhat organically from theinitial pure htools functionality.
The proposed organisation of the tree will be as follows:...
Implement compilation of regexes at creation time
This means that the verification of the correctness of the regex isdone once, at the deserialisation/creation time, as in the Pythoncode. To do this, we have to change the FilterRegex type from an alias...
Rename the Qlang and Queryd modules
Per the new query module hierarchy, rename Qlang to Query/Language andQueryd to Query/Server. This way, all query-related functionality isnow "contained" in the Query/ directory.
Add filtering support in Query
This adds basic infrastructure for filtering (fully functional except,as usual, for runtime data), and then uses it for node queries.
Since the filtering exports regex matching as an externalfunctionality, we have to use a regex library. There are many flavours...
Add an Applicative instance for our Result type
This will allow us to run generic computations in an Applicativecontext, if a monadic one is not needed (or not applicable due toclass constraints).
Parameterize the Filter type
In preparation for introducing filtering functionality, we convert the'Filter' type from a '*' kind to a '* -> *' kind.
This allows us to define some general properties for the filter, andfor example introduce later an easy filter compilation, etc....
Implement QueryFields for Nodes
Since we have all the definitions already, we can easily enablethis. Manual testing shows no difference between the Python and theHaskell versions of node list-fields.
Merge branch 'devel-2.6'
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Merge branch 'stable-2.6' into devel-2.6
Putting the multiallocate pieces together
This is the final part:
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Adding allocList function
This function iterates over the tryMGAlloc function, updates the nodelist and instance list, and refeeds it recursively until no newinstances are left.
This allows us then to allocate multiple instances and see if they wouldfit together....
Adding new IAllocator mode to constants
Fix warnings/errors with newer pylint
To help developing Ganeti on newer distributions, let's try to fixpylint warnings/errors. I'm using pylint from current Debian wheezy:pylint 0.25.1, astng 0.23.1, common 0.58.0, and we have 3 things thatneeds fixing....
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix decorator uses which crash newer pylint
Pylint version:
pylint 0.25.1, astng 0.23.1, common 0.58.0
crashes when passing the fully-qualified decorator name with:
File "/usr/lib/pymodules/python2.7/pylint/checkers/base.py", line 161, in visit_function...
Instance autorepair design
This design describes a tool that will perform automatic repairs oninstances when they are detected to be unhealthy (living on offline ordrained nodes, at the moment). These repairs can be scheduledautomatically or requested as a one-off by a tool or person....
Add missing luxi query 'QueryFields'
This was missed; we add the definition and the de-serialisationsupport.
Add Query support for Nodes (no filtering, no RPC)
This is the initial support for Query2: basic infrastructure (exceptfiltering) and node query support (without RPC).
It implements all the fields (tests by comparison with list-fields onthe Python side), except that:...
Move parameter title definitions to constants
This fixes an old TODO for moving these; now they can be reused in theHaskell Query2 implementation.
One more SADT type - VType
This adds the VTYPE_* conversion to VType, used for QFT equivalencelater.
Extend convert-constants support for dicts
This enhances convert-constants to not flatten dicts completely, butalso generate a so-called association list for them.
This allows either direct use of the 'lookup' function, or (forperformance) conversion to Data.Map and optimised lookup later....
Expand THH with more functionality for parameters
This adds two related improvements to THH:
- for parameters, we declare a list with all their fields, so that Query2 can build the list of fields (e.g. for hvparams, or ndparams) automatically
- we declare a new type class for "DictObjects", i.e. objects which...
Add more node-related data types and functions
This is a simple type declaration for NodeRole, a NdParamObject typeclass, and a few related helper functions for nodes and node groups.
Fix typo in node role description
I'd rather not introduce this typo in the Haskell code too, and alsonot having them identical is not good.
Add type classes for common objects
These mirror the TaggableObject in Python, in the sense that we willbe able to define generic functions for querying such fields.
Stub query2 call integration into QueryD
This patch corrects the definitions in Qlang.hs to match what Pythonexpects on the wire; this means replacing some manual data typedefinitions with 'buildObject' so that we get serialisation (and fieldnames) for free, adding (manually) JSON instances for types which are...
Fix error reporting for bad Luxi arguments in QueryD
Currently, the query daemon would simply close the connection to theclient without issuing a response, if parsing the arguments failed;the error was just logged.
Since this is very ugly from the client's point of view, we change it...
Improve error reporting in our JSON conversions
Reporting things such as the following in our error messages(indentation added by me, not originally present, so it's even worse):
JSArray [JSArray [JSRational False (1 % 1),JSString (JSONString {fromJSString = "a"})],...
Cleanup query socket before attempting to bind
This patch adds a simple function to ensure a socket doesn't existbefore binding to it. Additionally, we cleanup the exports ofQueryd.hs.
Complete and enable the haskell QueryClusterInfo call
Since we have now access to all cluster parameters, we can "fill" theparameters, and can finally enable it in CLI/RAPI.
Expand Objects.hs definitions
This patch adds the missing parameters in the cluster/group objects,for now as simple maps (dictionaries), without type safety. Therationale for adding them as such is:
- we need something to enable query functionality- since we don't modify the values, we don't risk introducing bugs...
Add a fillDict function
This is similar to the Python version, objects.py:FillDict.
Fix qualified import of Data.Map in QC.hs
As opposed to all other places in the code, we imported this asData.Map, leaving to longer names. Let's make it similar to the restof the code (import qualified Data.Map as Map).
Enhance convert-constants list generation
While looking at the hypervisor types in Constants.hs, I saw that the'hyperTypes' list is using strings instead of names. This is due tothe fact that we require the entire elements in the list to beidentified (homogeneous lists), but the string "xen-pvm" is declared...
Add two confd library tests
These test that we encode requests correctly and decode good messages,that a message with a too old/new timestamp is never accepted, andthat a message signed with a different key is never accepted.
Reduce some more code duplication and split code
The Qlang module defines ResultStatus, but it was already defined inGaneti/Luxi.hs; let's remove the duplicate definition from there sincethe proper place is in the newer module.
Also, in order to ease testing, we split some confd functions into a...
Add test for checking Haskell/Python opcode equivalence
This is a very big hack for testing the equivalence of Python andHaskell opcode definitions. See the docstring for details; I'm notvery happy with the solution but it does the job.
An alternate option would be to launch the Python code when...
Create a custom type for disk indices
While (again) trying to test Python/Haskell encoding interoperability,I found another bug: the disk index is declared in Python asht.TPositiveInt, but in Haskell just as Int, so it can take negativevalues too. Clearly we can do better, so let's add a wrapper type that...
Handle better 'null' values in optional fields
While testing Haskell⇔Python interoperability for opcodeserialisation, I found this bug: the Haskell code doesn't treatoptional fields with 'null' values as missing, which the Python codedoes, leading to differences....
Add test case for OpCode list equivalence
For now, we only test that we don't define extra opcodes in theHaskell code; once we have parity, we can enable the (for now)commented-out lines that check the reverse.
Add infrastructure for reading Python command output
This will be used in the future to make tests/assertions forPython/Haskell codebase equivalence (e.g. checking that the sameopcodes are defined, etc.).
(Side-note: this might not work in VPATH builds; I'll see how it goes)...
QCHelper: add support for defining HUnit test cases
This will allow us to use exactly the same method as for defining andusing QuickCheck properties. The differentiation is based on name,which is not very nice but is the same method used intest-framework-th, for example (so we will be able to switch...
Rename tests for nicer names in test output
With the change to test-framework, where individual tests are listed,QCHelper tries to remove a "prop_$group_" prefix from the functionnames, for shorter names. However, some test groups (LUXI) and testfunctions are not uniform, leading to long names, so we should fix...
Switch Haskell test harness to test-framework
This patch replaces our home-grown, and quite limited, test runnerinfrastructure with test-framework(http://batterseapower.github.com/test-framework/). The rationale fordoing so is as follows:
- we will need to add support for HUnit tests, so either we add more...
Simplify the Luxi client-server test
Since now we have the recvMsgExt, we can remove the handleEOFfunction.
Fix bug in the Luxi client-server test
We didn't close the server correctly, so for test sizes bigger thanthe max-open-files ulimit, the tests were failing in accept().
TH: one style fix and more docstrings
We were missing many docstrings in THH.hs, so let's add at least someof them, and fix some unquoted '/'. Additionally one style change hasbeen done.
OpCodes: build and export a list of all opcodes
This can be used for cross-checking with the Python code forconsistency on defined opcodes.
TH: Abstract function for computing constructor names
We'll need this in another place shortly, so let's abstract it and addproper verification of whether we were passed a type name correctly;the previous version would have failed with a pattern match failure,...
Enable tags query over the query socket
This patch adds the tags field to the objects which were missing it(all except Cluster), implements handling the LuxiCall QueryTags, andthen enables the use of the query socket in cli.ListTags, used by allcommands, and in the RAPI client....
Switch RAPI to use same tags listing as CLI
Currently, RAPI uses queries (as in QueryNode with field tags) forgetting the tags, whereas the CLI uses the QueryTags Luxicommand. Since this discrepancy is not good, let's switch RAPI to usethe same method as the CLI....
Config.hs: Add a function to lookup group
Like in the python code, this is a bit more complex since groups areindexed by UUID, so we must fallback to lookup via the name (slow; butthe number of groups should be small).
Add a Functor instance for our Result type
This will allow us to use the simpler applicative form (<$>, <*>)instead of monadic liftM, liftM2, etc.
Change the Luxi tags kind from String to a custom type
This will allow safer code when we implement the tags query.
Remove obsolete QrViaLuxi type
The actual query definitions are now in Qlang.hs, so let's use theItemType from there instead of luxi-defined type (which is alsoincomplete).
Implement support for query only clients in Rapi
This implements the same query=True|False functionality as inGetClient for cli.py, however since the RAPI code is much moreunit-tested (and the unit-test clients are mocked, for the most part,without support for addresses) we have to do many adaptations in the...
Add configure-time switch for split queries
And expand cli.GetClient() to allow opening the query socket, insteadof the main master socket.
Finally, enable the query socket use in gnt-cluster version, sincethat is already implemented fully in Queryd.hs/hconfd....
Remove obsolete conversion function for Luxi TH
Due to the (now removed) custom filter field, we needed a conversionfunction. Since now that field is gone, we can move to a simpler LuxiTH implementation.
Rename Query2.hs to Qlang.hs
While starting to use the new filter types, I realised that what iscurrently implemented is the equivalent of `lib/qlang.py', not`lib/query.py', since we only deal with data types for now and not theactual query runtime functionality (RPC, config, etc.)....
Switch Luxi Query operation to use a proper filter
Until now, since we didn't have a proper type for the encoded queryfilters, we were ignoring the filters and handled them as null values(JSNull and respectively ()). With the current Query2 implementation,...
Implement Query2 filter JSON (de)serialisation
This adds support for encoding/decoding Query2 filters to/from JSON,in (hopefully) the same format as the Python code generates.
It also adds a simple unit-test to check that this conversion isidempotent. Of note here is that, since the Filter data type is...
convert-constants: also export qlang.py constants
Since these will be needed for Haskell's implementation of query2.
Re-enable standard hlint warnings
Commit 5a1e31b4 (Add infrastructure for, and two extra hlint rules)was intended to add two extra hlint rules, but I didn't realise atthat time that "--hint" when first used overrides the built-inlints. As such, since then we were basically running with just those...
Run the query thread from confd
This enables the query functionality in confd.
Initial query daemon implementation
This is just a new module that exports a runQueryD function, that canbe imported to run a separate thread handling the luxi requests.
Currently it needs access just to the configuration, in the future itwill need access to an RPC runner too....
Enhance the Luxi interface implementation
This makes the implementation a bit nicer for both for server andclient side: we add a wrapper function with a better result type, anda few extra functions for building the response.
Add Objects definitions for the ispec/ipolicy types
Note that since we don't have yet a way to nicely handle two-leveloptional parameters, the Filled/Partial types and filling function areall manually built.
Improve Objects.hs definitions
This adds a few missing/incomplete definitions. We're still missingthe special parameters (disk params, hvparams, os_hvp).
Remove container field special cases
Since we now handle Containers uniformly, we can remove all traces ofthe special handling for this field type.
Improve the TH 'Container' type
This is the first part of the changes related to the 'Container' type.
We currently handle this type as follows: it's a simple type aliasover the Data.Map type, which means:
- it's easy to use the Data.Map functions to change the type...
Improve TH local variables naming
This patch addresses two issues with our TH code:
- using non-unique names (e.g. "std" for a local name, instead of "std_XXXX" random names), which can leads to conflicts; on the other hand, this makes the generated code a bit harder to parse...
Expand TH with tags field
Also add this new field and the other generic fields to the clusterobject.
Add new definition for query socket
This is not used yet, but is helpful when developing the Luxi endpointfor hconfd.
Simple QC tests for RPC calls
Right now we're only able to test if when a node is offline, the callfails with an appropriate errror.
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Implementation of NodeInfo call and result
node_info call takes hypervisors and vgs to ask for information aboutnode and returns bootid and results from hypervisors and volume groups.
Implementation of InstanceList call and result
instance_list call takes a list of hypervisors and returns a listof running instances.
Implementation of AllInstancesInfo call and result
all_instances_info call takes a list of hypervisors and returns a listof (name, memory, state, vcpus, time) - one element for eachinstance.
Infrastructure to execute RPC calls
Prepare and execute RPC call, prepare result for the call. Theseprocedures are generic in the sense that they only require theCall and Result types to be connected.We use curl library for sending http requests for RPC; as the library's...
Datatypes for haskell RPC calls
We introduce typeclasses for RPC call and result and create a typeclassthat binds the two together. For that we need to useMultiParamTypeClasses and FunctionalDependencies language pragmas, whichallow us to ensure that RPC result type can be deduced based on the...
Add Hypervisor and start exporting AdminState
Very simple Hypervisor object, that we want to pass in some of RPCcalls is added. We also export AdminState data type, as it is usedin one of the calls that will be implemented in this patch series.
Signed-off-by: Agata Murawska <agatamurawska@google.com>...
Move necessary constants from rpc & rpc_defs to constants
As timeout constants will now be used also on Haskell side, they havebeen moved from rpc_defs and rpc to constants. The same is true forconnection timeout for connecting to nodes.
Make stable-2.6 compatible with newer pep8
This is done so that all current branches can run with newer pep8;note that instead of fixing the problems (like I did on master), I'vejust silenced more. These should not be merged onto master!