History | View | Annotate | Download (49 kB)
Move htools backends to a separate directory
Five modules under the HTools/ directories are backendimplementations, so let's move them to a separate directory, to moreclearly show the hierarchy. I wanted to do this for a while, butmerging between branches is always an issue, so let's do it know since...
Add a simple unittest for THH code
This is very THH specific, and applies to all serialisations generatedby THH, so I'm adding it in its own module.
Probably we should add some more generic tests, but in general THHcode is tested by the various definitions; this new field type however...
Remove unused cache implementation
Note that this commit has no Makefile.am changes, as the files werenot actually used. So it's better to actually remove them.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add a default `.ghci' file
This options file for ghci preseeds the correct include paths, so thatinteractive sessions don't need to always pass these args.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add an Errors module mirroring the Python one
As described in the module doc string, while writing this it dawnedupon me that we're mixing all errors together into a single hierarchy(well, type on the Haskell side), which is not good. Some errors areused purely within noded, some in the CLI frontends, etc. so these...
Add small design for Linux HA integration
This documents the status (or wanted status) of some example ocf modulesI've written for Ganeti. They are far from perfect, but they should beshipped as a starting point for other people who want to run this to...
Replace LIBDIR in .in files
LIBDIR
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add initial implementation of prepare-node-join
This is a new tool as per the design document “design-ssh-setup”. Itreceives a JSON data structure on its standard input and configures theSSH daemon and root's SSH keys accordingly. Unit tests are included....
Find coverage during configure, issue with Debian package
- Debian Squeeze and up have a package named “python-coverage”, but it doesn't use the same binary name as upstream (“coverage”).- Said package includes a patch to use symlinks instead of file copies...
Add a new 'really-all' make target
This is used when one wants to build all binaries, including thosethat are used only for testing. A handy shortcut to make sure allbinaries can be built.
Add cluster monitoring agent design document
This design addresses the lack of a uniform way to query ganeti nodesfor real time information that can be used by monitoring.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge branch 'devel-2.6' into master
Add hbal-excl-tags.data to Makefile.am, missed in 0397694
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
gnt-job cancel: Confirmation and selection of jobs
New parameters, “--pending”, “--queued” and “--waiting”, are added toselect all jobs in the respective state. If one of those options is usedand “--force” is not given, the user is asked to confirm the operation....
Design for remote command execution via RPC
This is a first design for executing commands via RPC.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Makefile: Verify version of security.rst document
This document should be kept up-to-date.
Add a very simple test rpc program
This only supports test delay for now, is not built by default (onlyon demand), and is also not installed anywhere.
Add htools program code to hlint call
Currently only the libraries are added, but not the actual main binary(usually short) code.
Add design for changing node SSH setup
The goal is to remove the dependency on Paramiko.
Fix distcheck after commit 7a962ec
I only ran “make check” and forgot about “distcheck”.
Makefile/check-local: Ensure design document exists
This way a missing design document is detected.
Add design document for version 2.7
- Move design document for batch instance creation into it- Sort list of hidden files in index.rst
Cleanup haddock documentation a bit
While grepping for htools imports in the non-htools subdirectory, Isaw that our haddock prologue and title are very very old and refer tothe old htools-only state. Let's cleanup a bit…
Signed-off-by: Iustin Pop <iustin@google.com>...
Merge branch 'devel-2.6'
Merge branch 'stable-2.6' into devel-2.6
Improve Haskell configure options and detection
This patch cleans up the Haskell library detection and defaults.
First, it makes the base compiler/libraries required, per the email discussion.
It then adds two new small autoconf macros, on to check for a required...
Add list of design documents implemented in version 2.6
Each version should have its dedicated list.
Merge ganeti-master-cleaner back into ganeti-cleaner
As I wrote during/after the review on commit 2958c56, “ganeti-cleaner:Separate queue cleaning code”, while I appreciated the permissionseparation, I didn't like too much the file-based approach:
- it is a very simple script, and lots of the code is duplicated...
Enable bash completion for Haskell daemons too
This requires a few tiny changes to build-bash-completion and toMakefile rules, but is straightforward.
A few cleanups in Makefile.am
A lot of the lists in Makefile.am were not sorted properly (or atall); let's sort them for more sanity.
Additionally, check-local used to spew this big shell block, eventhough it does emit nice messages when failing, so we don't need to...
Fixup TAGS generation with newer GHC
Newer GHC refuses to allow "-O" with interactive mode, so let's filterthat out. Furthermore, sometimes you don't have a clean tree exactlywhen you need to look up something/update the tags, so let's filterout the "-Werror" too....
Rename Ganeti/HTools/Utils.hs to Ganeti/Utils.hs
This is, I believe, the last non-htools specific file that still livedin the htools directory; it's already widely used in non-htools code,so let's move it before we add more functionality to this module....
Design doc for partitioned Ganeti
Partitioned Ganeti allows to assign node resources in an exclusive manner,in order to limit interference between instances running on the same nodes.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Remove option to select the Python version of confd
This will need some more configure.ac checks for required libraries,which will come later with the Haskell dependencies reorganisation.
Remove the python confd server side code
In 2.7 we will only support the Haskell version, if enabled.
Since the original hconfd enabling was a bit hack-ish (copying overthe actual installed ganeti-confd, Python version), the Makefile.amchanges are a bit more involved than just the removal of the Python...
Add first version of virtual cluster documentation
- Add document describing virtual clusters - Non-root clusters are not yet described, will be done in another patch- Change title of design document to avoid confusion
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Always distribute ganeti.utils.io_unittest-runasroot.py
Tests that depend on fakeroot were not distributed in the tarball createdby "make dist" when fakeroot was missing. Now all tests are distributed,and a warning is printed when a test requires fakeroot but fakeroot is...
ganeti-cleaner: Separate queue cleaning code
This code does not need to run as root, therefore it's better to splitit out. It is now run with the same permissions as the master daemon.
Stop hardcoding root user
Some parts of the code still use a hardcoded user name: root. This patchreplaces all with a constant specified at build time. The end goal is tomake it possible to run a Ganeti cluster without any special privileges(of course this will prevent some functionality from working)....
Cleanup of build-time shell variable replacements
Instead of having a different set in (almost) every shell script, thisinserts the most commonly used variables at build time. This way thecode for injecting a root directory for virtual clusters also is just...
Fix distcheck after commit 29386d3ee
Missing backslash.
Add vcluster-setup utility
This utility can be used to configure an environment for a virtualcluster. It sets up entries in /etc/hosts, creates the necessarydirectory structure, and generates helper scripts.
Documentation for virtual clusters will come in a later patch....
NodeGroup query in Haskell
Implementation of nodegroup queries in Haskell. This is not yetcomplete as we are missing merged disk parameters and optionwant_diskparams is not implemented.
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Enable bash completion for htools scripts
This patch ties together the previous commits, by extendingbuild-bash-completion to call all htools scripts and get theircompletion information, then use that to build fake cli_optionsrepresenting them and finally generate the bash completion...
Add tests for bash completion
Given the number of settings that need to be controlled a plain bashscript was a better choice over shelltestrunner. Just a few completionsare attempted, but among them should be the most critical ones (e.g.“gnt-instance add --node …”....
Adding some fundamental unittests for iallocator
This test covers the bug fixes found in the previous two patches
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add new module for virtual clusters
This module will take care of managing paths for virtual clusters.Unittests are included (100% coverage).
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
bash_completion: Generate more compact version
First the numbers:$ stat --format '%s %n' doc/examples/bash_completion*77847 doc/examples/bash_completion86492 doc/examples/bash_completion-debug
The non-debug version doesn't use indentation and does not have some...
Use autoconf-based paths from Haskell instead of constants
Future changes will change Path.hs to use an environment variable.
constants: Move most paths to separate module
This is inpreparation for the implementation of virtual clusters. Manypaths will change based on an environment variable and are no longerconstant and should no longer be in “constants.py”. Since “constants.py”...
Refactor IAllocator code
The IAllocator class was handling all the requests on its own, passingin parameters on top level which works, but is hard to maintain and notflexible.
With the upcoming change to the IAllocator for MultiAllocate we can'tuse the toplevel parameters anymore. Therefore, we refactor the code...
Make the slots functionality more modular
As we will introduce another set of containers using the slots trickwe abstract away as much as possible to separate bases classes. Thechild classes then adapt them for their needs. This leads to less code...
Rework CLI modules and tests
While investigating how we could test the Daemon.hs module, I realisedthat we have a very, erm, sub-optimal situation:
- HTools/CLI.hs has a nice IO/pure separation testing in cmdline parsing, which allows some basic functionality to be tested, but...
Add query filter tests
These tests are node specific only because we don't have other querytypes implemented yet, but what they actually test is the variousfilter types.
The tests are trying to cover most filter functionality; missing fornow is proper checking for ContainsFilter and TrueFilter, the rest...
Add some unittests for node queries
These new tests check that:
- no known fields return unknown- any unknown field returns unknown- the type of the fields is consistent between the getters and the field definition- the length of each result row corresponds with the number of fields...
Add unittests for the BasicTypes module
This adds test properties for the various laws that the instances ofResult should follow; I could not find (offline) laws about `mappend',but otherwise I implemented all laws that I could find.
Note that we have to silence hlint warnings for the things we want to...
Move Version.hs up from under HTools/
This is another module that is generic, and not htools-specific.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Two tiny makefile fixes
During the recent moves and renames, two things have slipped through,since I didn't run make check-local…
Remove QC.hs and replace it with an auto-generated file
In order to have correct code coverage results, we must somehow importall production modules into the test runner. Until now, this was donemanually (when we didn't forget) in QC.hs.
To improve the situation, we remove QC.hs and replace it with an...
Split the Makefile variable HS_LIB_SRCS
This currently holds both production files and test files; since wewant to treat these separately, let's split the variable in two, andintroduce another one for the old, inclusive one.
Split last two remaining tests from QC.hs
This makes QC.hs obsolete, but we will remove it in a later patch,once we automatically build a file with all the imports (to ensurecorrect coverage results).
Move JSON.hs and Compat.hs out from under HTools/
These two files are not htools-specific, so let's move them out of theHTools subdirectory/module hierarchy and directly under Ganeti.
Split most HTools test code into separate files
Except for Ganeti.HTools.JSON, which needs rename, we split all theother test suites into separate files.
We have to add another common test helper, due to import dependencies(sigh), but otherwise this split is pretty straightforward....
Split Luxi, Qlang, Ssconf and OpCodes tests
… from QC.hs into their own files, again mirroring the production codesource tree.
Split Rpc tests from QC
This required lots of other code moves, so I created it as astandalone patch.
Split one more module out of QC and add test helpers
This splits the confd/utils tests, and adds the TestCommon module forshared test code.
Split out Objects.hs from QC.hs
This is the first file split out from QC.hs - an easy one, since ithas just one test.
The patch changes the way we build hpc excludes, since now we'll havemany modules that need to be excluded, and hpc doesn't seem to be able...
Move the unittest helper to a new Test/Ganeti dir
This patch starts the move of the test haskell code from`htools/Ganeti/HTools/' to its more proper place of`htest/Test/Ganeti'.
Move generated htools test stubs to htest/
This moves the last (I think) htools-related bits out of test/ underhtest/.
Move shelltests from test/ to htest/shelltests/
This is another rather trivial patch, moving all the (htools)shelltests to their own directory.
Move htools test files from test/data/htools to htest/data
Following up on the program moves, we now move the test data files.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Move haskell test code to htest/
This is the first commit of a series that will attempt to cleanup thetest code organisation, which evolved somewhat organically from theinitial pure htools functionality.
The proposed organisation of the tree will be as follows:...
Rename the Qlang and Queryd modules
Per the new query module hierarchy, rename Qlang to Query/Language andQueryd to Query/Server. This way, all query-related functionality isnow "contained" in the Query/ directory.
Add filtering support in Query
This adds basic infrastructure for filtering (fully functional except,as usual, for runtime data), and then uses it for node queries.
Since the filtering exports regex matching as an externalfunctionality, we have to use a regex library. There are many flavours...
Instance autorepair design
This design describes a tool that will perform automatic repairs oninstances when they are detected to be unhealthy (living on offline ordrained nodes, at the moment). These repairs can be scheduledautomatically or requested as a one-off by a tool or person....
Add Query support for Nodes (no filtering, no RPC)
This is the initial support for Query2: basic infrastructure (exceptfiltering) and node query support (without RPC).
It implements all the fields (tests by comparison with list-fields onthe Python side), except that:...
Stub query2 call integration into QueryD
This patch corrects the definitions in Qlang.hs to match what Pythonexpects on the wire; this means replacing some manual data typedefinitions with 'buildObject' so that we get serialisation (and fieldnames) for free, adding (manually) JSON instances for types which are...
Reduce some more code duplication and split code
The Qlang module defines ResultStatus, but it was already defined inGaneti/Luxi.hs; let's remove the duplicate definition from there sincethe proper place is in the newer module.
Also, in order to ease testing, we split some confd functions into a...
Add configure-time switch for split queries
And expand cli.GetClient() to allow opening the query socket, insteadof the main master socket.
Finally, enable the query socket use in gnt-cluster version, sincethat is already implemented fully in Queryd.hs/hconfd....
Rename Query2.hs to Qlang.hs
While starting to use the new filter types, I realised that what iscurrently implemented is the equivalent of `lib/qlang.py', not`lib/query.py', since we only deal with data types for now and not theactual query runtime functionality (RPC, config, etc.)....
Re-enable standard hlint warnings
Commit 5a1e31b4 (Add infrastructure for, and two extra hlint rules)was intended to add two extra hlint rules, but I didn't realise atthat time that "--hint" when first used overrides the built-inlints. As such, since then we were basically running with just those...
Initial query daemon implementation
This is just a new module that exports a runQueryD function, that canbe imported to run a separate thread handling the luxi requests.
Currently it needs access just to the configuration, in the future itwill need access to an RPC runner too....
Datatypes for haskell RPC calls
We introduce typeclasses for RPC call and result and create a typeclassthat binds the two together. For that we need to useMultiParamTypeClasses and FunctionalDependencies language pragmas, whichallow us to ensure that RPC result type can be deduced based on the...
Make stable-2.6 compatible with newer pep8
This is done so that all current branches can run with newer pep8;note that instead of fixing the problems (like I did on master), I'vejust silenced more. These should not be merged onto master!
Bump pep8 version to 1.2
Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:
- bump version in the docs- silence some new checks that are wrong due to our indent=2 instead of 4- fix lots of errors in the code where the indentation was wrong by 1...
Adding design doc for bulk instance create
This is the design doc for the bulk instance creation. You can moredetails in the doc itself.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Add new Makefile target to rebuild the whole dist
Due to the fact how the automake system works it doesn't rebuild alreadyprebuild files in distcheck. This lead to a bug, where a rebuild of thedocumentation was failing because we missed the fact that the files were...
Add missing rst files to Makefile.am
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Build epydoc.conf using standard replace_vars_sed
This is just begging to be converted to a standard replace_vars_sedrule, instead of custom sed calls.
Simplify some make rules
A rule of type "a/%: a/%.in" will also match "a/b/%: a/b/%.in", so noneed for the explicit examples/hooks rule. As for the man rules, theyare identical and thus can be collapsed.
We still have the problem that globally, not all our %.in to %...
Explicitly terminate some make rules
Generic rules like:
%: %.in
have the downside that the source (%.in) itself matches the target(via %.in: %.in.in). This leads to things like:
Looking for a rule with intermediate file `doc/examples/hooks/ipsec.in.in'....
Remove ancient implicit make rules
GNU Make contains some (ancient) implicit rules, that try toautomatically extract source files from RCS/SCCS version controlsystems. This is unneeded, and it pollutes the make -d outputsignificantly: after removing these rules (by defining empty targets...
Stop using BUILT_SOURCES
Commit dc7d2c49 introduced the use of BUILT_SOURCES to work aroundmissing dependencies. However, on closer reading of the gmake manual,BUILT_SOURCES is mainly used in cases where the dependencies are notknow before the build starts (e.g. with auto-generated C header...
Change how we create the 'ganeti' symlink
Currently, if one runs 'make' in an already fully-built tree, this isthe result:
cd . && test -h "ganeti" || { rm -f ganeti && ln -s lib ganeti; } make all-am make[1]: Entering directory `/tmp/test' cd . && test -h "ganeti" || { rm -f ganeti && ln -s lib ganeti; }...
Partial undo of "Makefile: Streamline directory creation"
Commit c964d962 changed the way we create directories, by two things:
- unifying all dependencies and ad-hoc directory creation into a single target (all_dirfiles)- changing how directories are created from a stamp file to .dir files...
A few style fixes in Makefile.am
Seen while debugging make rules.
Correctly do sub-$(MAKE) invocations
Reading the automake documentation, it seems that explicit calls to$(MAKE) need $(AM_MAKEFLAGS) passed, so let's do that.
Merge branch 'stable-2.6'