History | View | Annotate | Download (46.3 kB)
Add new module for virtual clusters
This module will take care of managing paths for virtual clusters.Unittests are included (100% coverage).
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
bash_completion: Generate more compact version
First the numbers:$ stat --format '%s %n' doc/examples/bash_completion*77847 doc/examples/bash_completion86492 doc/examples/bash_completion-debug
The non-debug version doesn't use indentation and does not have some...
Use autoconf-based paths from Haskell instead of constants
Future changes will change Path.hs to use an environment variable.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
constants: Move most paths to separate module
This is inpreparation for the implementation of virtual clusters. Manypaths will change based on an environment variable and are no longerconstant and should no longer be in “constants.py”. Since “constants.py”...
Refactor IAllocator code
The IAllocator class was handling all the requests on its own, passingin parameters on top level which works, but is hard to maintain and notflexible.
With the upcoming change to the IAllocator for MultiAllocate we can'tuse the toplevel parameters anymore. Therefore, we refactor the code...
Make the slots functionality more modular
As we will introduce another set of containers using the slots trickwe abstract away as much as possible to separate bases classes. Thechild classes then adapt them for their needs. This leads to less code...
Rework CLI modules and tests
While investigating how we could test the Daemon.hs module, I realisedthat we have a very, erm, sub-optimal situation:
- HTools/CLI.hs has a nice IO/pure separation testing in cmdline parsing, which allows some basic functionality to be tested, but...
Add query filter tests
These tests are node specific only because we don't have other querytypes implemented yet, but what they actually test is the variousfilter types.
The tests are trying to cover most filter functionality; missing fornow is proper checking for ContainsFilter and TrueFilter, the rest...
Add some unittests for node queries
These new tests check that:
- no known fields return unknown- any unknown field returns unknown- the type of the fields is consistent between the getters and the field definition- the length of each result row corresponds with the number of fields...
Add unittests for the BasicTypes module
This adds test properties for the various laws that the instances ofResult should follow; I could not find (offline) laws about `mappend',but otherwise I implemented all laws that I could find.
Note that we have to silence hlint warnings for the things we want to...
Move Version.hs up from under HTools/
This is another module that is generic, and not htools-specific.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Two tiny makefile fixes
During the recent moves and renames, two things have slipped through,since I didn't run make check-local…
Remove QC.hs and replace it with an auto-generated file
In order to have correct code coverage results, we must somehow importall production modules into the test runner. Until now, this was donemanually (when we didn't forget) in QC.hs.
To improve the situation, we remove QC.hs and replace it with an...
Split the Makefile variable HS_LIB_SRCS
This currently holds both production files and test files; since wewant to treat these separately, let's split the variable in two, andintroduce another one for the old, inclusive one.
Signed-off-by: Iustin Pop <iustin@google.com>...
Split last two remaining tests from QC.hs
This makes QC.hs obsolete, but we will remove it in a later patch,once we automatically build a file with all the imports (to ensurecorrect coverage results).
Move JSON.hs and Compat.hs out from under HTools/
These two files are not htools-specific, so let's move them out of theHTools subdirectory/module hierarchy and directly under Ganeti.
Split most HTools test code into separate files
Except for Ganeti.HTools.JSON, which needs rename, we split all theother test suites into separate files.
We have to add another common test helper, due to import dependencies(sigh), but otherwise this split is pretty straightforward....
Split Luxi, Qlang, Ssconf and OpCodes tests
… from QC.hs into their own files, again mirroring the production codesource tree.
Split Rpc tests from QC
This required lots of other code moves, so I created it as astandalone patch.
Split one more module out of QC and add test helpers
This splits the confd/utils tests, and adds the TestCommon module forshared test code.
Split out Objects.hs from QC.hs
This is the first file split out from QC.hs - an easy one, since ithas just one test.
The patch changes the way we build hpc excludes, since now we'll havemany modules that need to be excluded, and hpc doesn't seem to be able...
Move the unittest helper to a new Test/Ganeti dir
This patch starts the move of the test haskell code from`htools/Ganeti/HTools/' to its more proper place of`htest/Test/Ganeti'.
Move generated htools test stubs to htest/
This moves the last (I think) htools-related bits out of test/ underhtest/.
Move shelltests from test/ to htest/shelltests/
This is another rather trivial patch, moving all the (htools)shelltests to their own directory.
Move htools test files from test/data/htools to htest/data
Following up on the program moves, we now move the test data files.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Move haskell test code to htest/
This is the first commit of a series that will attempt to cleanup thetest code organisation, which evolved somewhat organically from theinitial pure htools functionality.
The proposed organisation of the tree will be as follows:...
Rename the Qlang and Queryd modules
Per the new query module hierarchy, rename Qlang to Query/Language andQueryd to Query/Server. This way, all query-related functionality isnow "contained" in the Query/ directory.
Add filtering support in Query
This adds basic infrastructure for filtering (fully functional except,as usual, for runtime data), and then uses it for node queries.
Since the filtering exports regex matching as an externalfunctionality, we have to use a regex library. There are many flavours...
Merge branch 'devel-2.6'
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Instance autorepair design
This design describes a tool that will perform automatic repairs oninstances when they are detected to be unhealthy (living on offline ordrained nodes, at the moment). These repairs can be scheduledautomatically or requested as a one-off by a tool or person....
Add Query support for Nodes (no filtering, no RPC)
This is the initial support for Query2: basic infrastructure (exceptfiltering) and node query support (without RPC).
It implements all the fields (tests by comparison with list-fields onthe Python side), except that:...
Stub query2 call integration into QueryD
This patch corrects the definitions in Qlang.hs to match what Pythonexpects on the wire; this means replacing some manual data typedefinitions with 'buildObject' so that we get serialisation (and fieldnames) for free, adding (manually) JSON instances for types which are...
Reduce some more code duplication and split code
The Qlang module defines ResultStatus, but it was already defined inGaneti/Luxi.hs; let's remove the duplicate definition from there sincethe proper place is in the newer module.
Also, in order to ease testing, we split some confd functions into a...
Add configure-time switch for split queries
And expand cli.GetClient() to allow opening the query socket, insteadof the main master socket.
Finally, enable the query socket use in gnt-cluster version, sincethat is already implemented fully in Queryd.hs/hconfd....
Rename Query2.hs to Qlang.hs
While starting to use the new filter types, I realised that what iscurrently implemented is the equivalent of `lib/qlang.py', not`lib/query.py', since we only deal with data types for now and not theactual query runtime functionality (RPC, config, etc.)....
Re-enable standard hlint warnings
Commit 5a1e31b4 (Add infrastructure for, and two extra hlint rules)was intended to add two extra hlint rules, but I didn't realise atthat time that "--hint" when first used overrides the built-inlints. As such, since then we were basically running with just those...
Initial query daemon implementation
This is just a new module that exports a runQueryD function, that canbe imported to run a separate thread handling the luxi requests.
Currently it needs access just to the configuration, in the future itwill need access to an RPC runner too....
Datatypes for haskell RPC calls
We introduce typeclasses for RPC call and result and create a typeclassthat binds the two together. For that we need to useMultiParamTypeClasses and FunctionalDependencies language pragmas, whichallow us to ensure that RPC result type can be deduced based on the...
Make stable-2.6 compatible with newer pep8
This is done so that all current branches can run with newer pep8;note that instead of fixing the problems (like I did on master), I'vejust silenced more. These should not be merged onto master!
Bump pep8 version to 1.2
Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:
- bump version in the docs- silence some new checks that are wrong due to our indent=2 instead of 4- fix lots of errors in the code where the indentation was wrong by 1...
Adding design doc for bulk instance create
This is the design doc for the bulk instance creation. You can moredetails in the doc itself.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Add new Makefile target to rebuild the whole dist
Due to the fact how the automake system works it doesn't rebuild alreadyprebuild files in distcheck. This lead to a bug, where a rebuild of thedocumentation was failing because we missed the fact that the files were...
Add missing rst files to Makefile.am
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Build epydoc.conf using standard replace_vars_sed
This is just begging to be converted to a standard replace_vars_sedrule, instead of custom sed calls.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Simplify some make rules
A rule of type "a/%: a/%.in" will also match "a/b/%: a/b/%.in", so noneed for the explicit examples/hooks rule. As for the man rules, theyare identical and thus can be collapsed.
We still have the problem that globally, not all our %.in to %...
Explicitly terminate some make rules
Generic rules like:
%: %.in
have the downside that the source (%.in) itself matches the target(via %.in: %.in.in). This leads to things like:
Looking for a rule with intermediate file `doc/examples/hooks/ipsec.in.in'....
Remove ancient implicit make rules
GNU Make contains some (ancient) implicit rules, that try toautomatically extract source files from RCS/SCCS version controlsystems. This is unneeded, and it pollutes the make -d outputsignificantly: after removing these rules (by defining empty targets...
Stop using BUILT_SOURCES
Commit dc7d2c49 introduced the use of BUILT_SOURCES to work aroundmissing dependencies. However, on closer reading of the gmake manual,BUILT_SOURCES is mainly used in cases where the dependencies are notknow before the build starts (e.g. with auto-generated C header...
Change how we create the 'ganeti' symlink
Currently, if one runs 'make' in an already fully-built tree, this isthe result:
cd . && test -h "ganeti" || { rm -f ganeti && ln -s lib ganeti; } make all-am make[1]: Entering directory `/tmp/test' cd . && test -h "ganeti" || { rm -f ganeti && ln -s lib ganeti; }...
Partial undo of "Makefile: Streamline directory creation"
Commit c964d962 changed the way we create directories, by two things:
- unifying all dependencies and ad-hoc directory creation into a single target (all_dirfiles)- changing how directories are created from a stamp file to .dir files...
A few style fixes in Makefile.am
Seen while debugging make rules.
Correctly do sub-$(MAKE) invocations
Reading the automake documentation, it seems that explicit calls to$(MAKE) need $(AM_MAKEFLAGS) passed, so let's do that.
Merge branch 'stable-2.6'
Fix Haskell coverage results
There are two current issues with the coverage values:
- we don't import all modules, thus leading to incomplete coverage results (too optimistic);- we use hpc in its default mode (intersection), which means that even modules which do have coverage results but are not used in all...
Also compute Haskell tags in "make tags"
Note however that this won't work correctly with older compilers, dueto ghc issue #4256; however, it's a handy way to build complete a TAGSfile for Emacs.
Cleanup the QC.hs file w.r.t. compiler options
Currently, we build the Haskell unittests with custom GHC flags,because we (I) were quite lazy when initially writing theunittests. This is not a problem for the tests themselves, but itcreates problem when (for example) one would want to pass all...
Add a Makefile rule to test unclean git status
This is trivial, but can be used easily from automated builds to checkthat the git tree is clean: files are not modified and untracked filesare not present.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Update time on Haskell binaries after build
This patch works around an issue in our build system. Since we don'tuse cabal or 'ghc -M', we don't track actual dependencies in ourMakefile; this in turn means that editing a file that only is used inthe main 'htools' binary will keep triggering 'ghc --make' for the...
Fix building of hcheck man page
The code used a space, not a tab, in Makefile.am, which means itactually broke the variable declaring the man pages.
Furthermore, we add some blank lines in the man page, otherwise theparagraphs will be run-in together....
Add a shell test for hbal and split instances
This is not perfect, as we only test that hbal completes successfullyand that it show a score improvement, but it's better than nothing.
Initial commit for introducting hcheck tool
Introduce infrastructure required to add Hcheck and build itsuccessfuly.
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
jstore: Add unittests
Add unittests for FormatJobID, GetArchiveDirectory, ParseJobId.
Haskell data types for query2
This contains the types to express query2 queries, filters and results.
Fix make -n use
Due to the way $(MAKE) behaves (it forces execution of the commandsthat contain it), use of $(MAKE) in a command line together withside-effects commands will break stuff. It's better to keep it eitheron a separate line, or if not possible, move said commands to a...
Enable hlint in lint and reorganise this target
Currently, the lint target does a sequential: pep8, pylint, pylint onthe QA sources. hlint is run only when explicitly requested.
This patch reorganises the lint target by:
- splitting the current lint target into separate pylint (slow),...
Remove one obsolete hlint override
During the preparations for 2.6, we cleaned up the codebase to usenewer exceptions (e.g. commits 30d25dd8, 79ac58fa), so we can now dropthis override.
Merge branch 'stable-2.5' into stable-2.6
Conflicts: Makefile.am (expected, from the parallel build fixes) qa/qa_config.py (due to cherry-pick, ignored, simply checkout out with --ours)...
Fix parallel build failures
This is the 2.5 version of the "fix build failures":
- man/%.gen could be left over even in case of failure, due to automake bug- make man/%.gen runs RUN_IN_TEMPDIR, so let's depend on it, since that target has the proper dependencies (create needed dirs)...
Add XEN_CMD to _autoconf.py
This is needed so we can specify the xen command upon configure time
Add checks from wrong hard-coded paths in manpages
This is trivial, but should prevent such things in the future.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reconcile Makefile.am and test data files
Sorry, forgot this in previous commit.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>(cherry picked from commit 1a1e7ab3f7cfe156152fb69961115a2c85b2a82d)
Workaround changed LVM behaviour
The vgreduce command has changed behaviour from when we initiallywrote the code (2.02.02 versus 2.02.66, 4 years delta):
- if there are LVs which will be impacted, it requires --force- otherwise refuses to proceed, but it still returns exit code 0...
Fix Makefile dependencies for building Constants.hs
I was wondering why my changes to luxi.py did not trigger the rebuild…
QA: Enable use of OR conditions in test checks
Until now “TestRunIf” and “TestEnabled” could only handle AND. With thispatch a new class named “Either” is added to “qa_config” and allows OR.The name “Either” was chosen instead of “Or” as the latter is very close...
Add example script for using RAPI test utilities
This script shows a few examples on how to use the RAPI input testclient. It is also run at “make check” time to ensure it's notcompletely broken.
QA: Add tests for “gnt-job list”
Update default instance kernel version
We switch from vmlinuz-2.6-… to vmlinuz-3-…. To do this nicely, wealso add a ./configure-time setting for the KVM instance kernel.
Add a special lexer for sphinx/pygments
This will be used throughout our docs for better formatting exampleshell sessions, with custom markup for comments, user fixed input anduser variable input.
Enable -Werror by default for htools
Since the code base is now "clean" across all supported GHC versions(6.12-7.4), we can enable -Werror again.
Add trivial tests for gnt-* cli
While testing some other stuff, I realised that the gnt-* commandscould be broken (as in, the script fails with syntax errors), but makecheck doesn't detect it. Since we have shelltest, we can now addtrivial tests for this case....
Enable selection between Python and Haskell confd
This patch changes configure.ac Makefile.am so that the user can pass:
- --disable-confd (or --enable-confd=no) to disable it completely- --enable-confd=yes or --enable-confd=python to select the traditional implementation (this is the default setting)...
Fix out-of-tree builds
The new shell tests do not succeed out-of-tree, due to static pathsand other issues. This trivial patch fixes these issue, make distchecknow passes.
Convert manual shell tests to shelltestrunner
This is more of a RFC… Basically most of the shell-based tests areconverted from exec+grep to shelltestrunner.
Things are not all fine and nice though:
- we have dependencies between tests, as some generate some data files...
Only build hconfd if --enable-confd was passed
A later, more complete patch, will allow selecting between either thePython version or the Haskell version. This is just a temporarysolution to help building without all the needed Haskell libraries.
Build test helpers that point to hpc-htools
Instead of using just shell constructs to run hpc-htools correctly(i.e. HTOOLS=role htools/hpc-htools …), let's add some shell fragmentsthat do this for us.
This will ease the running of tests directly.
Add skeleton ssconf module
This currently has only one export function in it, which will be usedfor future bind address functionality in daemons.
Rename htools-hpc to hpc-tools to fix laziness
This is a stupid/trivial patch. I didn't realise when addinghtools-hpc that this will "break" tab completion on htools; as in, Iwill have to add / manually always when completing files on thecommand line....
Add htools/rapi test directory to DIRS
Sorry!
Add tests for custom text files
Add RAPI shell-level unittest
Add new hail (shell) tests
Using some hand-crafted files (manually modified from a real clusterrunning master branch), we test that hail behaves as we expect:doesn't load obvious wrong data, can do relocations/allocations, etc.
Fix make distcheck after the new shell tests
With the new shell tests, we have for the first time the situationwhere tests require a binary built (via make), but the binary itselfis not the test. Hence, we need to declare and use check_SCRIPTS forit....
Build a coverage-enabled version of the tools
This allows the offline tests to also generate coverage data, and thehs-coverage target is changed to show both unit-test and offline-testscoverage.
The downside is that now we build yet-another-binary, which makes the...
Enable htools offline tests as Haskell tests
This enables the offline-test.sh run both in the special "hs-check" rule and during normal "make check".
htools: add confd daemon binary
This is currently not installed anywhere, just built.
htools: add confd server module
This contains a more-or-less complete implementation for theserver-side confd.
Note that most of the code is behaving identical to the Python code,with a notable exception: the asyncore/inotify code was changed (sinceHaskell doesn't have asyncore) to 3-thread system. While otherwise I...
htools: add basic daemon-related functionality
This is not complete for now, just the basic functionality has beenimplemented:
- daemonize- check we're running under the correct user- call setup logging
htools: add logging functionality
This adds a dependency on hslogger 1.1+, which is not available inDebian Squeeze, only in later versions.
htools: add runtime functionality
This duplicates the lib/runtime.py functionality, allowing us to checkfor runtime users and groups consistency.