History | View | Annotate | Download (49 kB)
Add new test for RAPI
Unlike existing tests, this actually tests RAPI at the interface withthe HTTP server. This way authentification can also be tested. A testfor “/2/query/…” is included as it's a bit special.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Automatically enable version-dependent GHC flags
Some GHC flags are very useful, but only appear in more recent GHCversions. To support the use of such flags while still supportingolder compilers, let's add conditional checks and enabling based onthe results....
Added attoparsec unit test for Unicode parsing
Attoparsec is known to have had issues with parsing non-ASCII strings.This test makes sure that parsing of Unicode characters works fine.
Signed-off-by: Michele Tartara <mtartara@google.com>[iustin: small doc string fixes]...
Improve make regen-vcs-version
"make regen-vcs-version" is run at every ./devel/upload time, in orderto have reasonably up-to-date information in the uploadedscripts/binaries. However, this means it currently triggersrecompilation of (at least) Ganeti/Version.hs, and sometimes more than...
Rename Confd.hs to Confd/Types.hs
This should be the last module rename, promise!
We rename this to conform to the other hierarchies (e.g. Query), andto not have both Confd.hs and Confd/*.hs.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Merge branch 'devel-2.6' into submit
Conflicts: Makefile.am (reordering, fixed) htools/Ganeti/Confd/Server.hs (hlint fixes on master) htools/Ganeti/Daemon.hs (hlint)...
Merge branch 'stable-2.6' into devel-2.6
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Switch devel/upload to a static file
We had twice in the past days questions about devel/upload being"broken", since bash re-reads shell scripts during their run and thisfile can get regenerated due to Makefile changes.
Since we only need this to be dynamically built for 3 variables, let's...
Fix gnt-instance console with xl
- Rename xm-console-wrapper to xen-console-wrapper- Pass the xen command to use as a parameter
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Remove setup-ssh
It has been superseeded by “prepare-node-join”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Move htools backends to a separate directory
Five modules under the HTools/ directories are backendimplementations, so let's move them to a separate directory, to moreclearly show the hierarchy. I wanted to do this for a while, butmerging between branches is always an issue, so let's do it know since...
Add a simple unittest for THH code
This is very THH specific, and applies to all serialisations generatedby THH, so I'm adding it in its own module.
Probably we should add some more generic tests, but in general THHcode is tested by the various definitions; this new field type however...
Remove unused cache implementation
Note that this commit has no Makefile.am changes, as the files werenot actually used. So it's better to actually remove them.
Add a default `.ghci' file
This options file for ghci preseeds the correct include paths, so thatinteractive sessions don't need to always pass these args.
Add an Errors module mirroring the Python one
As described in the module doc string, while writing this it dawnedupon me that we're mixing all errors together into a single hierarchy(well, type on the Haskell side), which is not good. Some errors areused purely within noded, some in the CLI frontends, etc. so these...
Add small design for Linux HA integration
This documents the status (or wanted status) of some example ocf modulesI've written for Ganeti. They are far from perfect, but they should beshipped as a starting point for other people who want to run this to...
Replace LIBDIR in .in files
LIBDIR
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add initial implementation of prepare-node-join
This is a new tool as per the design document “design-ssh-setup”. Itreceives a JSON data structure on its standard input and configures theSSH daemon and root's SSH keys accordingly. Unit tests are included....
Find coverage during configure, issue with Debian package
- Debian Squeeze and up have a package named “python-coverage”, but it doesn't use the same binary name as upstream (“coverage”).- Said package includes a patch to use symlinks instead of file copies...
Add a new 'really-all' make target
This is used when one wants to build all binaries, including thosethat are used only for testing. A handy shortcut to make sure allbinaries can be built.
Add cluster monitoring agent design document
This design addresses the lack of a uniform way to query ganeti nodesfor real time information that can be used by monitoring.
Merge branch 'devel-2.6' into master
Add hbal-excl-tags.data to Makefile.am, missed in 0397694
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
gnt-job cancel: Confirmation and selection of jobs
New parameters, “--pending”, “--queued” and “--waiting”, are added toselect all jobs in the respective state. If one of those options is usedand “--force” is not given, the user is asked to confirm the operation....
Design for remote command execution via RPC
This is a first design for executing commands via RPC.
Makefile: Verify version of security.rst document
This document should be kept up-to-date.
Add a very simple test rpc program
This only supports test delay for now, is not built by default (onlyon demand), and is also not installed anywhere.
Add htools program code to hlint call
Currently only the libraries are added, but not the actual main binary(usually short) code.
Add design for changing node SSH setup
The goal is to remove the dependency on Paramiko.
Fix distcheck after commit 7a962ec
I only ran “make check” and forgot about “distcheck”.
Makefile/check-local: Ensure design document exists
This way a missing design document is detected.
Add design document for version 2.7
- Move design document for batch instance creation into it- Sort list of hidden files in index.rst
Cleanup haddock documentation a bit
While grepping for htools imports in the non-htools subdirectory, Isaw that our haddock prologue and title are very very old and refer tothe old htools-only state. Let's cleanup a bit…
Signed-off-by: Iustin Pop <iustin@google.com>...
Merge branch 'devel-2.6'
Improve Haskell configure options and detection
This patch cleans up the Haskell library detection and defaults.
First, it makes the base compiler/libraries required, per the email discussion.
It then adds two new small autoconf macros, on to check for a required...
Add list of design documents implemented in version 2.6
Each version should have its dedicated list.
Merge ganeti-master-cleaner back into ganeti-cleaner
As I wrote during/after the review on commit 2958c56, “ganeti-cleaner:Separate queue cleaning code”, while I appreciated the permissionseparation, I didn't like too much the file-based approach:
- it is a very simple script, and lots of the code is duplicated...
Enable bash completion for Haskell daemons too
This requires a few tiny changes to build-bash-completion and toMakefile rules, but is straightforward.
A few cleanups in Makefile.am
A lot of the lists in Makefile.am were not sorted properly (or atall); let's sort them for more sanity.
Additionally, check-local used to spew this big shell block, eventhough it does emit nice messages when failing, so we don't need to...
Fixup TAGS generation with newer GHC
Newer GHC refuses to allow "-O" with interactive mode, so let's filterthat out. Furthermore, sometimes you don't have a clean tree exactlywhen you need to look up something/update the tags, so let's filterout the "-Werror" too....
Rename Ganeti/HTools/Utils.hs to Ganeti/Utils.hs
This is, I believe, the last non-htools specific file that still livedin the htools directory; it's already widely used in non-htools code,so let's move it before we add more functionality to this module....
Design doc for partitioned Ganeti
Partitioned Ganeti allows to assign node resources in an exclusive manner,in order to limit interference between instances running on the same nodes.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Remove option to select the Python version of confd
This will need some more configure.ac checks for required libraries,which will come later with the Haskell dependencies reorganisation.
Remove the python confd server side code
In 2.7 we will only support the Haskell version, if enabled.
Since the original hconfd enabling was a bit hack-ish (copying overthe actual installed ganeti-confd, Python version), the Makefile.amchanges are a bit more involved than just the removal of the Python...
Add first version of virtual cluster documentation
- Add document describing virtual clusters - Non-root clusters are not yet described, will be done in another patch- Change title of design document to avoid confusion
Always distribute ganeti.utils.io_unittest-runasroot.py
Tests that depend on fakeroot were not distributed in the tarball createdby "make dist" when fakeroot was missing. Now all tests are distributed,and a warning is printed when a test requires fakeroot but fakeroot is...
ganeti-cleaner: Separate queue cleaning code
This code does not need to run as root, therefore it's better to splitit out. It is now run with the same permissions as the master daemon.
Stop hardcoding root user
Some parts of the code still use a hardcoded user name: root. This patchreplaces all with a constant specified at build time. The end goal is tomake it possible to run a Ganeti cluster without any special privileges(of course this will prevent some functionality from working)....
Cleanup of build-time shell variable replacements
Instead of having a different set in (almost) every shell script, thisinserts the most commonly used variables at build time. This way thecode for injecting a root directory for virtual clusters also is just...
Fix distcheck after commit 29386d3ee
Missing backslash.
Add vcluster-setup utility
This utility can be used to configure an environment for a virtualcluster. It sets up entries in /etc/hosts, creates the necessarydirectory structure, and generates helper scripts.
Documentation for virtual clusters will come in a later patch....
NodeGroup query in Haskell
Implementation of nodegroup queries in Haskell. This is not yetcomplete as we are missing merged disk parameters and optionwant_diskparams is not implemented.
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Enable bash completion for htools scripts
This patch ties together the previous commits, by extendingbuild-bash-completion to call all htools scripts and get theircompletion information, then use that to build fake cli_optionsrepresenting them and finally generate the bash completion...
Add tests for bash completion
Given the number of settings that need to be controlled a plain bashscript was a better choice over shelltestrunner. Just a few completionsare attempted, but among them should be the most critical ones (e.g.“gnt-instance add --node …”....
Adding some fundamental unittests for iallocator
This test covers the bug fixes found in the previous two patches
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add new module for virtual clusters
This module will take care of managing paths for virtual clusters.Unittests are included (100% coverage).
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
bash_completion: Generate more compact version
First the numbers:$ stat --format '%s %n' doc/examples/bash_completion*77847 doc/examples/bash_completion86492 doc/examples/bash_completion-debug
The non-debug version doesn't use indentation and does not have some...
Use autoconf-based paths from Haskell instead of constants
Future changes will change Path.hs to use an environment variable.
constants: Move most paths to separate module
This is inpreparation for the implementation of virtual clusters. Manypaths will change based on an environment variable and are no longerconstant and should no longer be in “constants.py”. Since “constants.py”...
Refactor IAllocator code
The IAllocator class was handling all the requests on its own, passingin parameters on top level which works, but is hard to maintain and notflexible.
With the upcoming change to the IAllocator for MultiAllocate we can'tuse the toplevel parameters anymore. Therefore, we refactor the code...
Make the slots functionality more modular
As we will introduce another set of containers using the slots trickwe abstract away as much as possible to separate bases classes. Thechild classes then adapt them for their needs. This leads to less code...
Rework CLI modules and tests
While investigating how we could test the Daemon.hs module, I realisedthat we have a very, erm, sub-optimal situation:
- HTools/CLI.hs has a nice IO/pure separation testing in cmdline parsing, which allows some basic functionality to be tested, but...
Add query filter tests
These tests are node specific only because we don't have other querytypes implemented yet, but what they actually test is the variousfilter types.
The tests are trying to cover most filter functionality; missing fornow is proper checking for ContainsFilter and TrueFilter, the rest...
Add some unittests for node queries
These new tests check that:
- no known fields return unknown- any unknown field returns unknown- the type of the fields is consistent between the getters and the field definition- the length of each result row corresponds with the number of fields...
Add unittests for the BasicTypes module
This adds test properties for the various laws that the instances ofResult should follow; I could not find (offline) laws about `mappend',but otherwise I implemented all laws that I could find.
Note that we have to silence hlint warnings for the things we want to...
Move Version.hs up from under HTools/
This is another module that is generic, and not htools-specific.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Two tiny makefile fixes
During the recent moves and renames, two things have slipped through,since I didn't run make check-local…
Remove QC.hs and replace it with an auto-generated file
In order to have correct code coverage results, we must somehow importall production modules into the test runner. Until now, this was donemanually (when we didn't forget) in QC.hs.
To improve the situation, we remove QC.hs and replace it with an...
Split the Makefile variable HS_LIB_SRCS
This currently holds both production files and test files; since wewant to treat these separately, let's split the variable in two, andintroduce another one for the old, inclusive one.
Split last two remaining tests from QC.hs
This makes QC.hs obsolete, but we will remove it in a later patch,once we automatically build a file with all the imports (to ensurecorrect coverage results).
Move JSON.hs and Compat.hs out from under HTools/
These two files are not htools-specific, so let's move them out of theHTools subdirectory/module hierarchy and directly under Ganeti.
Split most HTools test code into separate files
Except for Ganeti.HTools.JSON, which needs rename, we split all theother test suites into separate files.
We have to add another common test helper, due to import dependencies(sigh), but otherwise this split is pretty straightforward....
Split Luxi, Qlang, Ssconf and OpCodes tests
… from QC.hs into their own files, again mirroring the production codesource tree.
Split Rpc tests from QC
This required lots of other code moves, so I created it as astandalone patch.
Split one more module out of QC and add test helpers
This splits the confd/utils tests, and adds the TestCommon module forshared test code.
Split out Objects.hs from QC.hs
This is the first file split out from QC.hs - an easy one, since ithas just one test.
The patch changes the way we build hpc excludes, since now we'll havemany modules that need to be excluded, and hpc doesn't seem to be able...
Move the unittest helper to a new Test/Ganeti dir
This patch starts the move of the test haskell code from`htools/Ganeti/HTools/' to its more proper place of`htest/Test/Ganeti'.
Move generated htools test stubs to htest/
This moves the last (I think) htools-related bits out of test/ underhtest/.
Move shelltests from test/ to htest/shelltests/
This is another rather trivial patch, moving all the (htools)shelltests to their own directory.
Move htools test files from test/data/htools to htest/data
Following up on the program moves, we now move the test data files.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Move haskell test code to htest/
This is the first commit of a series that will attempt to cleanup thetest code organisation, which evolved somewhat organically from theinitial pure htools functionality.
The proposed organisation of the tree will be as follows:...
Rename the Qlang and Queryd modules
Per the new query module hierarchy, rename Qlang to Query/Language andQueryd to Query/Server. This way, all query-related functionality isnow "contained" in the Query/ directory.
Add filtering support in Query
This adds basic infrastructure for filtering (fully functional except,as usual, for runtime data), and then uses it for node queries.
Since the filtering exports regex matching as an externalfunctionality, we have to use a regex library. There are many flavours...
Instance autorepair design
This design describes a tool that will perform automatic repairs oninstances when they are detected to be unhealthy (living on offline ordrained nodes, at the moment). These repairs can be scheduledautomatically or requested as a one-off by a tool or person....
Add Query support for Nodes (no filtering, no RPC)
This is the initial support for Query2: basic infrastructure (exceptfiltering) and node query support (without RPC).
It implements all the fields (tests by comparison with list-fields onthe Python side), except that:...
Stub query2 call integration into QueryD
This patch corrects the definitions in Qlang.hs to match what Pythonexpects on the wire; this means replacing some manual data typedefinitions with 'buildObject' so that we get serialisation (and fieldnames) for free, adding (manually) JSON instances for types which are...
Reduce some more code duplication and split code
The Qlang module defines ResultStatus, but it was already defined inGaneti/Luxi.hs; let's remove the duplicate definition from there sincethe proper place is in the newer module.
Also, in order to ease testing, we split some confd functions into a...
Add configure-time switch for split queries
And expand cli.GetClient() to allow opening the query socket, insteadof the main master socket.
Finally, enable the query socket use in gnt-cluster version, sincethat is already implemented fully in Queryd.hs/hconfd....
Rename Query2.hs to Qlang.hs
While starting to use the new filter types, I realised that what iscurrently implemented is the equivalent of `lib/qlang.py', not`lib/query.py', since we only deal with data types for now and not theactual query runtime functionality (RPC, config, etc.)....
Re-enable standard hlint warnings
Commit 5a1e31b4 (Add infrastructure for, and two extra hlint rules)was intended to add two extra hlint rules, but I didn't realise atthat time that "--hint" when first used overrides the built-inlints. As such, since then we were basically running with just those...
Initial query daemon implementation
This is just a new module that exports a runQueryD function, that canbe imported to run a separate thread handling the luxi requests.
Currently it needs access just to the configuration, in the future itwill need access to an RPC runner too....
Datatypes for haskell RPC calls
We introduce typeclasses for RPC call and result and create a typeclassthat binds the two together. For that we need to useMultiParamTypeClasses and FunctionalDependencies language pragmas, whichallow us to ensure that RPC result type can be deduced based on the...
Make stable-2.6 compatible with newer pep8
This is done so that all current branches can run with newer pep8;note that instead of fixing the problems (like I did on master), I'vejust silenced more. These should not be merged onto master!
Bump pep8 version to 1.2
Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:
- bump version in the docs- silence some new checks that are wrong due to our indent=2 instead of 4- fix lots of errors in the code where the indentation was wrong by 1...
Adding design doc for bulk instance create
This is the design doc for the bulk instance creation. You can moredetails in the doc itself.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Add new Makefile target to rebuild the whole dist
Due to the fact how the automake system works it doesn't rebuild alreadyprebuild files in distcheck. This lead to a bug, where a rebuild of thedocumentation was failing because we missed the fact that the files were...
Add missing rst files to Makefile.am
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>