History | View | Annotate | Download (51.5 kB)
Add version check for doc/design-draft.rst
This is similar to README or “doc/security.rst”. Having a comment thatneeds updating for a new version is more likely going to trigger a checkof the lists of design drafts.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Makefile/check-local: List all errors before failing
Use the “error” variable in multiple places and fail only once all testshave been completed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix a bug in the hs-coverage rule of Makefile.am
The coverage information for mon-collector was not merged correctly.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add shelltests for the mon-collector
This commit adds shelltests for the mon-collector binary and for the DRBDdata collector.
Also, it fixes a small bug in the DRBD parser found thanks to the tests.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Add the stand-alone data collector for DRBD
It is implemented as a single binary receiving, as its first parameter,the name of the actual data collector to be run.This way, its structure can be used for all the future data collectors.
Also, factored out of bdev.py into constants.py the location of the DRBD...
Network and address pool (again)
Implementation of the network objects and address pool. Functionalityas in the reverted commit b9a616e132af9bba718d2da8c94eeec7af886814, butnow using only the vector library and not the bit-vector library.Tested with vector library version 0.9 and 0.10., which are...
Make htools/Ganeti/Constants.hs depend on errors.py
It also uses values from “lib/errors.py”.
Revert "Network and address pool"
This reverts commit b9a616e132af9bba718d2da8c94eeec7af886814, whichdepends on the "bitvec" library. We need to investigate a bit further thatdependency, as it in turns depends on "vector" 0.9.1 or below, since0.10.* removed support for Data.Vector.Unboxed.Safe which it uses....
Network and address pool
Implementation of the network and address pool class inHaskell. Not complete yet. Includes unit tests that coverall functionality that is so far implemented.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add tool to clean up node
Sometimes a node is not removed properly from a cluster (especiallyduring development). This new tool stops all daemons and removes (aftermaking copies) the most critical files.
Add tool to configure node daemon
The design for this is in “doc/design-node-add.rst”. The tool receives aJSON data structure on stdin and configures the node's daemon afterverifying the received values.
Makefile: Reset environment for epydoc
epydoc allows overriding configuration values via environment variables.While this might be useful in certain use cases, but as no prefixwhatsoever is used, conflicts are easily created. Some people have theenvironment variable “NAME” set, effectively overriding the project name...
Add Ganeti.HTools.Graph
This module implements some algorithms on Data.Graph data structures.At the moment its main functionality is an LF-color implementation(greedy coloring in descending order of degree). There are also a fewextra functions to calculate the degree order, and convert the node to...
Small Makefile.am cleanups
Indentation within rules is fixed to always be 2 spaces (as opposed tomixed tabs/2 spaces/4 spaces). Additionally, the check-dirs rule,which was more complex than necessary, has been simplified: movedauto4mte.cache to DIRCHECK_EXCLUDE, and simplify the find call and the...
Fix long lines in Ganeti.HTools.Types and add Make check
Since hlint doesn't check style issues but rather code only issues,let's add a simple Makefile check for too-long lines, and fix thesingle bad case we have currently.
Signed-off-by: Iustin Pop <iustin@google.com>...
Two small haskell-related Makefile improvements
Just some cleanup:
- don't run hlint over hpc-htools, since it's just a symlink to htools.hs (so we'd get duplicate warnings)- build rpc-test in make really-all, via HS_ALL_PROGS variable
Improve the Haskell api doc directory creation system
Instead of having two different lists of directories, now there is only one.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Design for opportunistic locking
Add the design document for opportunistic locking.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add unit tests for the JSON serialization of DRBD status
The serialization itself is done by Text.JSON, so the tests deal with checkingthat Text.JSON objects are created correctly from the DRBD parser datastructures.
Fix Haskell profiling targets
It turns out that in newer GHC versions, the suffix shouldn't beprefixed with a dot; otherwise name mangling is broken.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Fix Attoparsec test coverage check compatibility
The hpc tool, on squeeze, does not handle utf8 encoded files correctly.The Attoparsec test is just needed to verify whether the parser handlesuft8 correctly, so the solution is to exclude it from coverage checks....
ssconf: Verify file size when reading, add some tests
Until now ssconf would limit the amount read from files to 128 KiB andsilently ignored files larger than that. With this patch a check isadded by using fstat(2) on the file descriptor while it's being read....
Rename configure option for restricted commands
It is no longer “--enable-remote-commands”, but rather“--enable-restricted-commands”.
Include networks in Makefile and bash completion
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Iustin Pop <iustin@google.com>
Add DRBD parser unit tests
This adds tests that existing test files can be parsed by the Haskellparser as well, plus one new test file.
Add parser for DRBD /proc file
Split OpCode.hs and add module for opcode parameters
Due to TemplateHaskell stage restrictions, we can't define parametersin the same module as we're using them for TH, so we have to defineall module parameters in a separate module.
This patch therefore splits OpCodes.hs in two, adding that module and...
Create a new Ganeti/Types.hs module
There are already three cases where we copied type definitions betweenthe htools-specific types into the main ganeti code. Let's stop doingthis ☺ and create a common types module that holds these.
Note that there already exists BasicTypes.hs, but that refers to very...
configure: Shell cleanup
- Remote unnecessary quote characters- Change some uses of “test "$foo"” to “test n "$foo"” Merge “if/else/if” into “if/elif”- Wrap lines longer than 80 characters and add a test to “make check”
Prevent hlint crash on unicode chars
Some versions of hlint crash on the Attoparsec test file because theyare not able to correctly handle unicode characters if the proper encoding(--utf8) is not specified.
Rename design document for SSH setup
More design details will be added regarding setting up the node daemon,so this is more about adding a node than just setting up SSH.
backend: Implement remote commands
As per design document (doc/design-remote-commands.rst), a number ofrather strict tests is applied to any incoming request, a delay isinserted upon errors and returned error messages are very generic(unless it's the actual command that failed). There are unit tests for...
configure: Add option to enable remote commands
By default remote commands are disabled and need to be explicitelyenabled at build time.
Add ganeti-node-role ocf example file
This allows offlining nodes that don't respond if they are part of alinux-HA cluster.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add ganeti-master-role.ocf example file
This allows controlling the cluster master role if the nodes are part ofa linux-HA cluster.
Add new test for RAPI
Unlike existing tests, this actually tests RAPI at the interface withthe HTTP server. This way authentification can also be tested. A testfor “/2/query/…” is included as it's a bit special.
Automatically enable version-dependent GHC flags
Some GHC flags are very useful, but only appear in more recent GHCversions. To support the use of such flags while still supportingolder compilers, let's add conditional checks and enabling based onthe results....
Added attoparsec unit test for Unicode parsing
Attoparsec is known to have had issues with parsing non-ASCII strings.This test makes sure that parsing of Unicode characters works fine.
Signed-off-by: Michele Tartara <mtartara@google.com>[iustin: small doc string fixes]...
Improve make regen-vcs-version
"make regen-vcs-version" is run at every ./devel/upload time, in orderto have reasonably up-to-date information in the uploadedscripts/binaries. However, this means it currently triggersrecompilation of (at least) Ganeti/Version.hs, and sometimes more than...
Rename Confd.hs to Confd/Types.hs
This should be the last module rename, promise!
We rename this to conform to the other hierarchies (e.g. Query), andto not have both Confd.hs and Confd/*.hs.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Merge branch 'devel-2.6' into submit
Conflicts: Makefile.am (reordering, fixed) htools/Ganeti/Confd/Server.hs (hlint fixes on master) htools/Ganeti/Daemon.hs (hlint)...
Merge branch 'stable-2.6' into devel-2.6
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Switch devel/upload to a static file
We had twice in the past days questions about devel/upload being"broken", since bash re-reads shell scripts during their run and thisfile can get regenerated due to Makefile changes.
Since we only need this to be dynamically built for 3 variables, let's...
Fix gnt-instance console with xl
- Rename xm-console-wrapper to xen-console-wrapper- Pass the xen command to use as a parameter
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Remove setup-ssh
It has been superseeded by “prepare-node-join”.
Move htools backends to a separate directory
Five modules under the HTools/ directories are backendimplementations, so let's move them to a separate directory, to moreclearly show the hierarchy. I wanted to do this for a while, butmerging between branches is always an issue, so let's do it know since...
Add a simple unittest for THH code
This is very THH specific, and applies to all serialisations generatedby THH, so I'm adding it in its own module.
Probably we should add some more generic tests, but in general THHcode is tested by the various definitions; this new field type however...
Remove unused cache implementation
Note that this commit has no Makefile.am changes, as the files werenot actually used. So it's better to actually remove them.
Add a default `.ghci' file
This options file for ghci preseeds the correct include paths, so thatinteractive sessions don't need to always pass these args.
Add an Errors module mirroring the Python one
As described in the module doc string, while writing this it dawnedupon me that we're mixing all errors together into a single hierarchy(well, type on the Haskell side), which is not good. Some errors areused purely within noded, some in the CLI frontends, etc. so these...
Add small design for Linux HA integration
This documents the status (or wanted status) of some example ocf modulesI've written for Ganeti. They are far from perfect, but they should beshipped as a starting point for other people who want to run this to...
Replace LIBDIR in .in files
LIBDIR
Add initial implementation of prepare-node-join
This is a new tool as per the design document “design-ssh-setup”. Itreceives a JSON data structure on its standard input and configures theSSH daemon and root's SSH keys accordingly. Unit tests are included....
Find coverage during configure, issue with Debian package
- Debian Squeeze and up have a package named “python-coverage”, but it doesn't use the same binary name as upstream (“coverage”).- Said package includes a patch to use symlinks instead of file copies...
Add a new 'really-all' make target
This is used when one wants to build all binaries, including thosethat are used only for testing. A handy shortcut to make sure allbinaries can be built.
Add cluster monitoring agent design document
This design addresses the lack of a uniform way to query ganeti nodesfor real time information that can be used by monitoring.
Merge branch 'devel-2.6' into master
Add hbal-excl-tags.data to Makefile.am, missed in 0397694
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
gnt-job cancel: Confirmation and selection of jobs
New parameters, “--pending”, “--queued” and “--waiting”, are added toselect all jobs in the respective state. If one of those options is usedand “--force” is not given, the user is asked to confirm the operation....
Design for remote command execution via RPC
This is a first design for executing commands via RPC.
Makefile: Verify version of security.rst document
This document should be kept up-to-date.
Add a very simple test rpc program
This only supports test delay for now, is not built by default (onlyon demand), and is also not installed anywhere.
Add htools program code to hlint call
Currently only the libraries are added, but not the actual main binary(usually short) code.
Add design for changing node SSH setup
The goal is to remove the dependency on Paramiko.
Fix distcheck after commit 7a962ec
I only ran “make check” and forgot about “distcheck”.
Makefile/check-local: Ensure design document exists
This way a missing design document is detected.
Add design document for version 2.7
- Move design document for batch instance creation into it- Sort list of hidden files in index.rst
Cleanup haddock documentation a bit
While grepping for htools imports in the non-htools subdirectory, Isaw that our haddock prologue and title are very very old and refer tothe old htools-only state. Let's cleanup a bit…
Merge branch 'devel-2.6'
Improve Haskell configure options and detection
This patch cleans up the Haskell library detection and defaults.
First, it makes the base compiler/libraries required, per the email discussion.
It then adds two new small autoconf macros, on to check for a required...
Add list of design documents implemented in version 2.6
Each version should have its dedicated list.
Merge ganeti-master-cleaner back into ganeti-cleaner
As I wrote during/after the review on commit 2958c56, “ganeti-cleaner:Separate queue cleaning code”, while I appreciated the permissionseparation, I didn't like too much the file-based approach:
- it is a very simple script, and lots of the code is duplicated...
Enable bash completion for Haskell daemons too
This requires a few tiny changes to build-bash-completion and toMakefile rules, but is straightforward.
A few cleanups in Makefile.am
A lot of the lists in Makefile.am were not sorted properly (or atall); let's sort them for more sanity.
Additionally, check-local used to spew this big shell block, eventhough it does emit nice messages when failing, so we don't need to...
Fixup TAGS generation with newer GHC
Newer GHC refuses to allow "-O" with interactive mode, so let's filterthat out. Furthermore, sometimes you don't have a clean tree exactlywhen you need to look up something/update the tags, so let's filterout the "-Werror" too....
Rename Ganeti/HTools/Utils.hs to Ganeti/Utils.hs
This is, I believe, the last non-htools specific file that still livedin the htools directory; it's already widely used in non-htools code,so let's move it before we add more functionality to this module....
Design doc for partitioned Ganeti
Partitioned Ganeti allows to assign node resources in an exclusive manner,in order to limit interference between instances running on the same nodes.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Remove the python confd server side code
In 2.7 we will only support the Haskell version, if enabled.
Since the original hconfd enabling was a bit hack-ish (copying overthe actual installed ganeti-confd, Python version), the Makefile.amchanges are a bit more involved than just the removal of the Python...
Remove option to select the Python version of confd
This will need some more configure.ac checks for required libraries,which will come later with the Haskell dependencies reorganisation.
Add first version of virtual cluster documentation
- Add document describing virtual clusters - Non-root clusters are not yet described, will be done in another patch- Change title of design document to avoid confusion
Always distribute ganeti.utils.io_unittest-runasroot.py
Tests that depend on fakeroot were not distributed in the tarball createdby "make dist" when fakeroot was missing. Now all tests are distributed,and a warning is printed when a test requires fakeroot but fakeroot is...
ganeti-cleaner: Separate queue cleaning code
This code does not need to run as root, therefore it's better to splitit out. It is now run with the same permissions as the master daemon.
Stop hardcoding root user
Some parts of the code still use a hardcoded user name: root. This patchreplaces all with a constant specified at build time. The end goal is tomake it possible to run a Ganeti cluster without any special privileges(of course this will prevent some functionality from working)....
Cleanup of build-time shell variable replacements
Instead of having a different set in (almost) every shell script, thisinserts the most commonly used variables at build time. This way thecode for injecting a root directory for virtual clusters also is just...
Fix distcheck after commit 29386d3ee
Missing backslash.
Add vcluster-setup utility
This utility can be used to configure an environment for a virtualcluster. It sets up entries in /etc/hosts, creates the necessarydirectory structure, and generates helper scripts.
Documentation for virtual clusters will come in a later patch....
NodeGroup query in Haskell
Implementation of nodegroup queries in Haskell. This is not yetcomplete as we are missing merged disk parameters and optionwant_diskparams is not implemented.
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Enable bash completion for htools scripts
This patch ties together the previous commits, by extendingbuild-bash-completion to call all htools scripts and get theircompletion information, then use that to build fake cli_optionsrepresenting them and finally generate the bash completion...
Add tests for bash completion
Given the number of settings that need to be controlled a plain bashscript was a better choice over shelltestrunner. Just a few completionsare attempted, but among them should be the most critical ones (e.g.“gnt-instance add --node …”....
Adding some fundamental unittests for iallocator
This test covers the bug fixes found in the previous two patches
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add new module for virtual clusters
This module will take care of managing paths for virtual clusters.Unittests are included (100% coverage).
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
bash_completion: Generate more compact version
First the numbers:$ stat --format '%s %n' doc/examples/bash_completion*77847 doc/examples/bash_completion86492 doc/examples/bash_completion-debug
The non-debug version doesn't use indentation and does not have some...
Use autoconf-based paths from Haskell instead of constants
Future changes will change Path.hs to use an environment variable.
constants: Move most paths to separate module
This is inpreparation for the implementation of virtual clusters. Manypaths will change based on an environment variable and are no longerconstant and should no longer be in “constants.py”. Since “constants.py”...
Refactor IAllocator code
The IAllocator class was handling all the requests on its own, passingin parameters on top level which works, but is hard to maintain and notflexible.
With the upcoming change to the IAllocator for MultiAllocate we can'tuse the toplevel parameters anymore. Therefore, we refactor the code...
Make the slots functionality more modular
As we will introduce another set of containers using the slots trickwe abstract away as much as possible to separate bases classes. Thechild classes then adapt them for their needs. This leads to less code...
Rework CLI modules and tests
While investigating how we could test the Daemon.hs module, I realisedthat we have a very, erm, sub-optimal situation:
- HTools/CLI.hs has a nice IO/pure separation testing in cmdline parsing, which allows some basic functionality to be tested, but...
Add query filter tests
These tests are node specific only because we don't have other querytypes implemented yet, but what they actually test is the variousfilter types.
The tests are trying to cover most filter functionality; missing fornow is proper checking for ContainsFilter and TrueFilter, the rest...
Add some unittests for node queries
These new tests check that:
- no known fields return unknown- any unknown field returns unknown- the type of the fields is consistent between the getters and the field definition- the length of each result row corresponds with the number of fields...