History | View | Annotate | Download (47.9 kB)
Enable bash completion for Haskell daemons too
This requires a few tiny changes to build-bash-completion and toMakefile rules, but is straightforward.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
A few cleanups in Makefile.am
A lot of the lists in Makefile.am were not sorted properly (or atall); let's sort them for more sanity.
Additionally, check-local used to spew this big shell block, eventhough it does emit nice messages when failing, so we don't need to...
Fixup TAGS generation with newer GHC
Newer GHC refuses to allow "-O" with interactive mode, so let's filterthat out. Furthermore, sometimes you don't have a clean tree exactlywhen you need to look up something/update the tags, so let's filterout the "-Werror" too....
Rename Ganeti/HTools/Utils.hs to Ganeti/Utils.hs
This is, I believe, the last non-htools specific file that still livedin the htools directory; it's already widely used in non-htools code,so let's move it before we add more functionality to this module....
Design doc for partitioned Ganeti
Partitioned Ganeti allows to assign node resources in an exclusive manner,in order to limit interference between instances running on the same nodes.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Remove the python confd server side code
In 2.7 we will only support the Haskell version, if enabled.
Since the original hconfd enabling was a bit hack-ish (copying overthe actual installed ganeti-confd, Python version), the Makefile.amchanges are a bit more involved than just the removal of the Python...
Remove option to select the Python version of confd
This will need some more configure.ac checks for required libraries,which will come later with the Haskell dependencies reorganisation.
Add first version of virtual cluster documentation
- Add document describing virtual clusters - Non-root clusters are not yet described, will be done in another patch- Change title of design document to avoid confusion
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Always distribute ganeti.utils.io_unittest-runasroot.py
Tests that depend on fakeroot were not distributed in the tarball createdby "make dist" when fakeroot was missing. Now all tests are distributed,and a warning is printed when a test requires fakeroot but fakeroot is...
ganeti-cleaner: Separate queue cleaning code
This code does not need to run as root, therefore it's better to splitit out. It is now run with the same permissions as the master daemon.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Stop hardcoding root user
Some parts of the code still use a hardcoded user name: root. This patchreplaces all with a constant specified at build time. The end goal is tomake it possible to run a Ganeti cluster without any special privileges(of course this will prevent some functionality from working)....
Cleanup of build-time shell variable replacements
Instead of having a different set in (almost) every shell script, thisinserts the most commonly used variables at build time. This way thecode for injecting a root directory for virtual clusters also is just...
Fix distcheck after commit 29386d3ee
Missing backslash.
Add vcluster-setup utility
This utility can be used to configure an environment for a virtualcluster. It sets up entries in /etc/hosts, creates the necessarydirectory structure, and generates helper scripts.
Documentation for virtual clusters will come in a later patch....
NodeGroup query in Haskell
Implementation of nodegroup queries in Haskell. This is not yetcomplete as we are missing merged disk parameters and optionwant_diskparams is not implemented.
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Enable bash completion for htools scripts
This patch ties together the previous commits, by extendingbuild-bash-completion to call all htools scripts and get theircompletion information, then use that to build fake cli_optionsrepresenting them and finally generate the bash completion...
Add tests for bash completion
Given the number of settings that need to be controlled a plain bashscript was a better choice over shelltestrunner. Just a few completionsare attempted, but among them should be the most critical ones (e.g.“gnt-instance add --node …”....
Adding some fundamental unittests for iallocator
This test covers the bug fixes found in the previous two patches
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add new module for virtual clusters
This module will take care of managing paths for virtual clusters.Unittests are included (100% coverage).
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
bash_completion: Generate more compact version
First the numbers:$ stat --format '%s %n' doc/examples/bash_completion*77847 doc/examples/bash_completion86492 doc/examples/bash_completion-debug
The non-debug version doesn't use indentation and does not have some...
Use autoconf-based paths from Haskell instead of constants
Future changes will change Path.hs to use an environment variable.
constants: Move most paths to separate module
This is inpreparation for the implementation of virtual clusters. Manypaths will change based on an environment variable and are no longerconstant and should no longer be in “constants.py”. Since “constants.py”...
Refactor IAllocator code
The IAllocator class was handling all the requests on its own, passingin parameters on top level which works, but is hard to maintain and notflexible.
With the upcoming change to the IAllocator for MultiAllocate we can'tuse the toplevel parameters anymore. Therefore, we refactor the code...
Make the slots functionality more modular
As we will introduce another set of containers using the slots trickwe abstract away as much as possible to separate bases classes. Thechild classes then adapt them for their needs. This leads to less code...
Rework CLI modules and tests
While investigating how we could test the Daemon.hs module, I realisedthat we have a very, erm, sub-optimal situation:
- HTools/CLI.hs has a nice IO/pure separation testing in cmdline parsing, which allows some basic functionality to be tested, but...
Add query filter tests
These tests are node specific only because we don't have other querytypes implemented yet, but what they actually test is the variousfilter types.
The tests are trying to cover most filter functionality; missing fornow is proper checking for ContainsFilter and TrueFilter, the rest...
Add some unittests for node queries
These new tests check that:
- no known fields return unknown- any unknown field returns unknown- the type of the fields is consistent between the getters and the field definition- the length of each result row corresponds with the number of fields...
Add unittests for the BasicTypes module
This adds test properties for the various laws that the instances ofResult should follow; I could not find (offline) laws about `mappend',but otherwise I implemented all laws that I could find.
Note that we have to silence hlint warnings for the things we want to...
Move Version.hs up from under HTools/
This is another module that is generic, and not htools-specific.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Two tiny makefile fixes
During the recent moves and renames, two things have slipped through,since I didn't run make check-local…
Remove QC.hs and replace it with an auto-generated file
In order to have correct code coverage results, we must somehow importall production modules into the test runner. Until now, this was donemanually (when we didn't forget) in QC.hs.
To improve the situation, we remove QC.hs and replace it with an...
Split the Makefile variable HS_LIB_SRCS
This currently holds both production files and test files; since wewant to treat these separately, let's split the variable in two, andintroduce another one for the old, inclusive one.
Signed-off-by: Iustin Pop <iustin@google.com>...
Split last two remaining tests from QC.hs
This makes QC.hs obsolete, but we will remove it in a later patch,once we automatically build a file with all the imports (to ensurecorrect coverage results).
Move JSON.hs and Compat.hs out from under HTools/
These two files are not htools-specific, so let's move them out of theHTools subdirectory/module hierarchy and directly under Ganeti.
Split most HTools test code into separate files
Except for Ganeti.HTools.JSON, which needs rename, we split all theother test suites into separate files.
We have to add another common test helper, due to import dependencies(sigh), but otherwise this split is pretty straightforward....
Split Luxi, Qlang, Ssconf and OpCodes tests
… from QC.hs into their own files, again mirroring the production codesource tree.
Split Rpc tests from QC
This required lots of other code moves, so I created it as astandalone patch.
Split one more module out of QC and add test helpers
This splits the confd/utils tests, and adds the TestCommon module forshared test code.
Split out Objects.hs from QC.hs
This is the first file split out from QC.hs - an easy one, since ithas just one test.
The patch changes the way we build hpc excludes, since now we'll havemany modules that need to be excluded, and hpc doesn't seem to be able...
Move the unittest helper to a new Test/Ganeti dir
This patch starts the move of the test haskell code from`htools/Ganeti/HTools/' to its more proper place of`htest/Test/Ganeti'.
Move generated htools test stubs to htest/
This moves the last (I think) htools-related bits out of test/ underhtest/.
Move shelltests from test/ to htest/shelltests/
This is another rather trivial patch, moving all the (htools)shelltests to their own directory.
Move htools test files from test/data/htools to htest/data
Following up on the program moves, we now move the test data files.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Move haskell test code to htest/
This is the first commit of a series that will attempt to cleanup thetest code organisation, which evolved somewhat organically from theinitial pure htools functionality.
The proposed organisation of the tree will be as follows:...
Rename the Qlang and Queryd modules
Per the new query module hierarchy, rename Qlang to Query/Language andQueryd to Query/Server. This way, all query-related functionality isnow "contained" in the Query/ directory.
Add filtering support in Query
This adds basic infrastructure for filtering (fully functional except,as usual, for runtime data), and then uses it for node queries.
Since the filtering exports regex matching as an externalfunctionality, we have to use a regex library. There are many flavours...
Merge branch 'devel-2.6'
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Instance autorepair design
This design describes a tool that will perform automatic repairs oninstances when they are detected to be unhealthy (living on offline ordrained nodes, at the moment). These repairs can be scheduledautomatically or requested as a one-off by a tool or person....
Add Query support for Nodes (no filtering, no RPC)
This is the initial support for Query2: basic infrastructure (exceptfiltering) and node query support (without RPC).
It implements all the fields (tests by comparison with list-fields onthe Python side), except that:...
Stub query2 call integration into QueryD
This patch corrects the definitions in Qlang.hs to match what Pythonexpects on the wire; this means replacing some manual data typedefinitions with 'buildObject' so that we get serialisation (and fieldnames) for free, adding (manually) JSON instances for types which are...
Reduce some more code duplication and split code
The Qlang module defines ResultStatus, but it was already defined inGaneti/Luxi.hs; let's remove the duplicate definition from there sincethe proper place is in the newer module.
Also, in order to ease testing, we split some confd functions into a...
Add configure-time switch for split queries
And expand cli.GetClient() to allow opening the query socket, insteadof the main master socket.
Finally, enable the query socket use in gnt-cluster version, sincethat is already implemented fully in Queryd.hs/hconfd....
Rename Query2.hs to Qlang.hs
While starting to use the new filter types, I realised that what iscurrently implemented is the equivalent of `lib/qlang.py', not`lib/query.py', since we only deal with data types for now and not theactual query runtime functionality (RPC, config, etc.)....
Re-enable standard hlint warnings
Commit 5a1e31b4 (Add infrastructure for, and two extra hlint rules)was intended to add two extra hlint rules, but I didn't realise atthat time that "--hint" when first used overrides the built-inlints. As such, since then we were basically running with just those...
Initial query daemon implementation
This is just a new module that exports a runQueryD function, that canbe imported to run a separate thread handling the luxi requests.
Currently it needs access just to the configuration, in the future itwill need access to an RPC runner too....
Datatypes for haskell RPC calls
We introduce typeclasses for RPC call and result and create a typeclassthat binds the two together. For that we need to useMultiParamTypeClasses and FunctionalDependencies language pragmas, whichallow us to ensure that RPC result type can be deduced based on the...
Make stable-2.6 compatible with newer pep8
This is done so that all current branches can run with newer pep8;note that instead of fixing the problems (like I did on master), I'vejust silenced more. These should not be merged onto master!
Bump pep8 version to 1.2
Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:
- bump version in the docs- silence some new checks that are wrong due to our indent=2 instead of 4- fix lots of errors in the code where the indentation was wrong by 1...
Adding design doc for bulk instance create
This is the design doc for the bulk instance creation. You can moredetails in the doc itself.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Add new Makefile target to rebuild the whole dist
Due to the fact how the automake system works it doesn't rebuild alreadyprebuild files in distcheck. This lead to a bug, where a rebuild of thedocumentation was failing because we missed the fact that the files were...
Add missing rst files to Makefile.am
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Build epydoc.conf using standard replace_vars_sed
This is just begging to be converted to a standard replace_vars_sedrule, instead of custom sed calls.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Simplify some make rules
A rule of type "a/%: a/%.in" will also match "a/b/%: a/b/%.in", so noneed for the explicit examples/hooks rule. As for the man rules, theyare identical and thus can be collapsed.
We still have the problem that globally, not all our %.in to %...
Explicitly terminate some make rules
Generic rules like:
%: %.in
have the downside that the source (%.in) itself matches the target(via %.in: %.in.in). This leads to things like:
Looking for a rule with intermediate file `doc/examples/hooks/ipsec.in.in'....
Remove ancient implicit make rules
GNU Make contains some (ancient) implicit rules, that try toautomatically extract source files from RCS/SCCS version controlsystems. This is unneeded, and it pollutes the make -d outputsignificantly: after removing these rules (by defining empty targets...
Stop using BUILT_SOURCES
Commit dc7d2c49 introduced the use of BUILT_SOURCES to work aroundmissing dependencies. However, on closer reading of the gmake manual,BUILT_SOURCES is mainly used in cases where the dependencies are notknow before the build starts (e.g. with auto-generated C header...
Change how we create the 'ganeti' symlink
Currently, if one runs 'make' in an already fully-built tree, this isthe result:
cd . && test -h "ganeti" || { rm -f ganeti && ln -s lib ganeti; } make all-am make[1]: Entering directory `/tmp/test' cd . && test -h "ganeti" || { rm -f ganeti && ln -s lib ganeti; }...
Partial undo of "Makefile: Streamline directory creation"
Commit c964d962 changed the way we create directories, by two things:
- unifying all dependencies and ad-hoc directory creation into a single target (all_dirfiles)- changing how directories are created from a stamp file to .dir files...
A few style fixes in Makefile.am
Seen while debugging make rules.
Correctly do sub-$(MAKE) invocations
Reading the automake documentation, it seems that explicit calls to$(MAKE) need $(AM_MAKEFLAGS) passed, so let's do that.
Merge branch 'stable-2.6'
Fix Haskell coverage results
There are two current issues with the coverage values:
- we don't import all modules, thus leading to incomplete coverage results (too optimistic);- we use hpc in its default mode (intersection), which means that even modules which do have coverage results but are not used in all...
Also compute Haskell tags in "make tags"
Note however that this won't work correctly with older compilers, dueto ghc issue #4256; however, it's a handy way to build complete a TAGSfile for Emacs.
Cleanup the QC.hs file w.r.t. compiler options
Currently, we build the Haskell unittests with custom GHC flags,because we (I) were quite lazy when initially writing theunittests. This is not a problem for the tests themselves, but itcreates problem when (for example) one would want to pass all...
Add a Makefile rule to test unclean git status
This is trivial, but can be used easily from automated builds to checkthat the git tree is clean: files are not modified and untracked filesare not present.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Update time on Haskell binaries after build
This patch works around an issue in our build system. Since we don'tuse cabal or 'ghc -M', we don't track actual dependencies in ourMakefile; this in turn means that editing a file that only is used inthe main 'htools' binary will keep triggering 'ghc --make' for the...
Fix building of hcheck man page
The code used a space, not a tab, in Makefile.am, which means itactually broke the variable declaring the man pages.
Furthermore, we add some blank lines in the man page, otherwise theparagraphs will be run-in together....
Add a shell test for hbal and split instances
This is not perfect, as we only test that hbal completes successfullyand that it show a score improvement, but it's better than nothing.
Initial commit for introducting hcheck tool
Introduce infrastructure required to add Hcheck and build itsuccessfuly.
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
jstore: Add unittests
Add unittests for FormatJobID, GetArchiveDirectory, ParseJobId.
Haskell data types for query2
This contains the types to express query2 queries, filters and results.
Fix make -n use
Due to the way $(MAKE) behaves (it forces execution of the commandsthat contain it), use of $(MAKE) in a command line together withside-effects commands will break stuff. It's better to keep it eitheron a separate line, or if not possible, move said commands to a...
Enable hlint in lint and reorganise this target
Currently, the lint target does a sequential: pep8, pylint, pylint onthe QA sources. hlint is run only when explicitly requested.
This patch reorganises the lint target by:
- splitting the current lint target into separate pylint (slow),...
Remove one obsolete hlint override
During the preparations for 2.6, we cleaned up the codebase to usenewer exceptions (e.g. commits 30d25dd8, 79ac58fa), so we can now dropthis override.
Merge branch 'stable-2.5' into stable-2.6
Conflicts: Makefile.am (expected, from the parallel build fixes) qa/qa_config.py (due to cherry-pick, ignored, simply checkout out with --ours)...
Fix parallel build failures
This is the 2.5 version of the "fix build failures":
- man/%.gen could be left over even in case of failure, due to automake bug- make man/%.gen runs RUN_IN_TEMPDIR, so let's depend on it, since that target has the proper dependencies (create needed dirs)...
Add XEN_CMD to _autoconf.py
This is needed so we can specify the xen command upon configure time
Add checks from wrong hard-coded paths in manpages
This is trivial, but should prevent such things in the future.
Reconcile Makefile.am and test data files
Sorry, forgot this in previous commit.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>(cherry picked from commit 1a1e7ab3f7cfe156152fb69961115a2c85b2a82d)
Workaround changed LVM behaviour
The vgreduce command has changed behaviour from when we initiallywrote the code (2.02.02 versus 2.02.66, 4 years delta):
- if there are LVs which will be impacted, it requires --force- otherwise refuses to proceed, but it still returns exit code 0...
Fix Makefile dependencies for building Constants.hs
I was wondering why my changes to luxi.py did not trigger the rebuild…
QA: Enable use of OR conditions in test checks
Until now “TestRunIf” and “TestEnabled” could only handle AND. With thispatch a new class named “Either” is added to “qa_config” and allows OR.The name “Either” was chosen instead of “Or” as the latter is very close...
Add example script for using RAPI test utilities
This script shows a few examples on how to use the RAPI input testclient. It is also run at “make check” time to ensure it's notcompletely broken.
QA: Add tests for “gnt-job list”
Update default instance kernel version
We switch from vmlinuz-2.6-… to vmlinuz-3-…. To do this nicely, wealso add a ./configure-time setting for the KVM instance kernel.
Add a special lexer for sphinx/pygments
This will be used throughout our docs for better formatting exampleshell sessions, with custom markup for comments, user fixed input anduser variable input.
Enable -Werror by default for htools
Since the code base is now "clean" across all supported GHC versions(6.12-7.4), we can enable -Werror again.
Add trivial tests for gnt-* cli
While testing some other stuff, I realised that the gnt-* commandscould be broken (as in, the script fails with syntax errors), but makecheck doesn't detect it. Since we have shelltest, we can now addtrivial tests for this case....
Enable selection between Python and Haskell confd
This patch changes configure.ac Makefile.am so that the user can pass:
- --disable-confd (or --enable-confd=no) to disable it completely- --enable-confd=yes or --enable-confd=python to select the traditional implementation (this is the default setting)...