History | View | Annotate | Download (56 kB)
First part of Network Queries in Haskell
This is the beginning of the implementation of networkqueries. This includes establishing all infrastructureto run the network queries and implement querying ofsome simpler fields and the node group listing.
Signed-off-by: Helga Velroyen <helgav@google.com>...
hv_xen: Refactor getting node information, add tests
Refactor and add tests for getting node (Domain-0) information.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
hv_xen: Refactor running & parsing "xm list", add tests
This patch refactors “_RunXmList” and adds some tests.
Make Xen config path a build-time option
Stop hardcoding the path in “hv_xen.py”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Makefile.am: Protect placeholders for sed
If the placeholder has exactly the same name as the variable used inautoconf it would be replaced as well. In the case of“ENABLE_SPLIT_QUERY” this led to the sed command “s#False#False#g”.Inserting additional quotes after the first @ sign prevents this....
ENABLE_SPLIT_QUERY
Makefile: Allow tarball creation when man-in-doc is active
Documentation with included man pages should not be distributed as itcontains fixed paths. Since the documentation with man pages is actuallywritten to a separate directory, tarball creation is still possible....
run-in-tempdir: Copy doc/* only if needed
To preserve disk space on copying all files to a temporary directory,the “doc” directory is only copied when building documentation. Acomment describing the need is also added to Makefile.am, as well as asuperfluous direct invocation of Bash removed....
Add reporting infrastructure for data collectors
This commit adds the part of the JSON report generation code that will becommon to all the data collectors, according to the format specified in thedesign document.
Signed-off-by: Michele Tartara <mtartara@google.com>...
burnin: Add unittest for supported disk templates
When new a new disk template is added, burnin should be updated.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Move burnin to ganeti.tools.burnin
Similar to other scripts, “tools/burnin” just becomes a wrapper script.
Link man pages in documentation
This patch depends on “Option to include man pages in documentation”. Inthe documentation build including man pages, all “:manpage:`…`”references are converted to links. For man pages not provided by Ganeti,Sphinx' standard formatting is used....
Option to include man pages in documentation
Before this patch, HTML versions of man pages (man/*.rst) were alreadybuilt. However, since they are separate from the normal documentation,their content is not indexed for Sphinx' search functionality.Additionally it would simply be nice to have everything in one place....
kvm: add two help dump files
One is the --help output for kvm 1.1.2 and the second one is the same as0.9.1 but with a fake option added between -drive and its boot=on|offoption: this tests that if boot=on|off appears in another option after-drive it is not accepted by our regexp....
Generate documentation from build directory
When man pages should be included they need to be copied from man/*.rst.This means documentation can no longer be built from the static reSTfiles alone (which are referenced using “abs_top_srcdir”). Similar to...
Add script to check man page references
This script checks for some of the most obvious mistakes when formattingman page references (which should have the form “**ganeti**\(7)”). Whilethis works now, it is very hard to avoid ambiguities (e.g. references...
Makefile: Fix list of directories
Commit dbee5c9 had a wrong path.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Highlight external links in HTML documentation
Based on MediaWiki's monobook skin.
Unit tests for bdev and utils.lvm
Also amended a comment of a tested method.The tests are for methods introduced in commit 63c73073 (LVM disk creationuses dedicated PVs).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Support instance-minor pairing in the DRBD collector
This commits enables the DRBD data collector to use the Confd client togather information about the pairing between DRBD minors and instances.
For testing purposes, the DRBD data collector now requires either zero...
Add footer.rst as a dependency for all man pages
Commit 6aff051 removed an intermediate step, but forgot to add“footer.rst” as a direct dependency.
Remove intermediate footer.man/footer.html files
I don't remember why I introduced these when switching to pandoc, butthey are not absolutely needed. Instead of pre-generating them (whichmight save some trivial cpu time), we can simply pass footer.rst as an...
Enable a table of contents for the html version of man pages
This makes the (very dull) html version of the man pages a bit morereadable, until such a time that we can switch over to sphinx fully.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Stop manually building images out of dot files
The first dot files we added way back in the 2.0 release are managedmanually: we build the .pngs ourselves, track them in .gitingore,etc. Since then, we have switched to allow sphinx to build/manage themitself, so it makes sense to cleanup the build system and covert...
Inline the standalone dot files
This removes entirely the standalone dot files by inlining them in theRST documents - as they weren't referenced from more than one documenteither way.
Add small webserver for development
When working on documentation it can be helpful to use a browser. Insome environments it's not possible to access the files directly from agraphical browser. This trivial webserver exports all files in thecurrent directory and unlike alternatives such as thttpd, doesn't check...
Makefile.am: Add missing space
Commit acaf3ca missed this space before a backslash.
Fix the installation location of mon-collector
The mon-collector binary was erroneously installed twice: once in/usr/bin/ and once in /usr/lib/ganeti/.With this modification, it is now installed in /usr/lib/ganeti/ only,as intended.
In order to do this in a proper way, some parts of the Makefile.am where...
Move src/Ganeti/HTools/Program.hs to Program/Main.hs
This removes one more tab conflict; this is the last module in ourcode where we have both x.hs and x/.
Furthermore, we collapse all actual code into the new Main.hs module,leaving the htools.hs basically empty (will allow better testing in...
A bit more name cleanup
- gen-coverage → gen-py-coverage- coverage-htools.tix → coverage-hs.tix
Split the hs-check test rule
Currently, both the unit test and the shell tests are run (manually)from this rule. This has two downsides: you can't (compile and) runjust one of them easily, and make can't run them in parallel.
By splitting the rules, we can both of the above easily....
Rename test/hs/test to htest
The current base name of 'test' means that this binary will generateprofiling, coverage, etc. files with the name 'test.*', result intab-completion conflicts with the test directory. Let's slightlychange its base name for typing easiness....
Move htest/ files under the test/ tree
htest/data becomes test/data/htools (basically reverting commit8feabc89), and htest/* becomes test/hs/*.
Most changes beside the rename are trivial s/…/…, with the exceptionof autotools/run-in-tempdir, which needed some more changes now that...
Move python test files to test/py
This is the first step of the test files reorganisation: moving test/*(except test/data) to new directory test/py/.
Replace htools→hs in configure/makefile rules
This continues the cleanup of the 'htools' name, by replacing 'htools'with 'hs' in configure and makefile rules.
There is still 'WANT_HTOOLS' left as it is correct, but some of theconditionals in Makefile.am will need some cleanup (in a different...
Fix dependency on built python sources for hs-check
Since we call into Python for opcodes and other type compatibilitychecks, we need to ensure that the python sources arebuilt. Otherwise, full parallel make might or might not break,depending on timing....
Rename htools/ to src/
Per offline discussions, this is the first patch of therenames. Tested with "make distcheck", seems to work fine.
The only change outside of the renaming is a bit of simplification inthe .gitignore rules; otherwise, simply s/htools/src/....
Fix building of Constants.hs w.r.t .pyc files
Unless this is run under run-in-tempdir, it will create as usual the.pyc files.
Add the gnt-storage client
Add a new client called 'gnt-storage'.The client interacts with the ExtStorage interface, similarly tothe way gnt-os interacts with the OS interface.
For now, only two commands are supported: 'info' and 'diagnose'.
'diagnose' calculates the node status of each provider on each node,...
Add ExtStorage related man pages
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Extend the hcheck shell tests
This adds a few new hcheck tests, to extend the coverage.
Extend the hbal shell tests
After this patch, almost all of hbal except the actual Luxi executionis being covered by unit- or shelltests.
Fix build after addition of 'empty-cluster.data'
Of course I forgot to add this to TEST_FILES… and only distcheckcatches it.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Added class to contain information about a PV
This makes the code more readable and easier to upgrade.bdev.LogicalVolume.GetPVInfo and the code that depends on it have beenrefactored to use the new class.
utils.CheckVolumeGroupSize() has been moved to lib/utils/lvm.py, where more...
Add Confd client to the Haskell code base
The client queries all the master candidates in parallel, until the minimumnumber of replies, defined in the constant file, is received.A timeout prevents the waiting from being of indefinite length.
The reply to be returned to the function that made the query is decided...
Implement the External Storage Interface
With this commit we introduce the External Storage Interfaceto Ganeti, abbreviated: ExtStorage Interface.
The ExtStorage Interface provides Ganeti with the ability to interactwith externally connected shared storage pools, visible by all...
Add hroller htools personality
This is a new personality that for the moment doesn't do anything.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add support for job queries in hconfd
This adds support for job queries, including (basic) unit-tests.
I've tested this for memory and cpu usage as follows:
- 3600 jobs (live queue): - via masterd, default: ~1.1s (masterd: ~60MB ram) - via confd, default: ~1.1s (hconfd: ~25MB ram)...
Add a read-only job queue module
This patch adds implementation for a read-only job queue module,together with "full" test (as full as can be in a lazy language…).
One note about the behaviour of the job queue is the handling ofopcodes that fail validation: the 'input' opcode actually is a...
Enable bash completion for mon-collector
This just ties in all the pieces so far and enables bash completionfor it.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add more mon-collector shell tests
These test the newly-changed behaviour, and fix an inconsistency inthe hs-check target (versus hs-coverage).
Remove two hlint overrides
"Use comparing" was not needed, and "Use on" can be solved by actuallyusing on in the single case we have (the patch also reorders importsin that file).
Add a custom CLI module for DataCollectors
Currently, the mon-collector binary uses the HTools/CLI module, whichis OK but mean it links in lots of htools code. By copying that moduleto DataCollectors/CLI and removing the unneeded code, we reduce thenumber of modules it depends on fro 20 to 12, meaning both a shorter...
Fix profiling targets
As noted by Guido, there are problems when using the hs-prof andhs-prof-quick targets in the default configuration (compiling allprograms). The errors manifest in the form of wrong symbols duringcompilation.
I knew that the hs-prof targets, which compiled multiple objects with...
Split and extend Confd types tests
Some confd types were not tested for serialisation, so let's movethese (type-related) tests to their own module and extend them.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Export jstore constants as well to Constants.hs
This single constant could be as well moved to constants.py. For now Ichoose to just export the module as well.
Add version check for doc/design-draft.rst
This is similar to README or “doc/security.rst”. Having a comment thatneeds updating for a new version is more likely going to trigger a checkof the lists of design drafts.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Makefile/check-local: List all errors before failing
Use the “error” variable in multiple places and fail only once all testshave been completed.
Fix a bug in the hs-coverage rule of Makefile.am
The coverage information for mon-collector was not merged correctly.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add shelltests for the mon-collector
This commit adds shelltests for the mon-collector binary and for the DRBDdata collector.
Also, it fixes a small bug in the DRBD parser found thanks to the tests.
Add the stand-alone data collector for DRBD
It is implemented as a single binary receiving, as its first parameter,the name of the actual data collector to be run.This way, its structure can be used for all the future data collectors.
Also, factored out of bdev.py into constants.py the location of the DRBD...
Network and address pool (again)
Implementation of the network objects and address pool. Functionalityas in the reverted commit b9a616e132af9bba718d2da8c94eeec7af886814, butnow using only the vector library and not the bit-vector library.Tested with vector library version 0.9 and 0.10., which are...
Make htools/Ganeti/Constants.hs depend on errors.py
It also uses values from “lib/errors.py”.
Revert "Network and address pool"
This reverts commit b9a616e132af9bba718d2da8c94eeec7af886814, whichdepends on the "bitvec" library. We need to investigate a bit further thatdependency, as it in turns depends on "vector" 0.9.1 or below, since0.10.* removed support for Data.Vector.Unboxed.Safe which it uses....
Network and address pool
Implementation of the network and address pool class inHaskell. Not complete yet. Includes unit tests that coverall functionality that is so far implemented.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add tool to clean up node
Sometimes a node is not removed properly from a cluster (especiallyduring development). This new tool stops all daemons and removes (aftermaking copies) the most critical files.
Add tool to configure node daemon
The design for this is in “doc/design-node-add.rst”. The tool receives aJSON data structure on stdin and configures the node's daemon afterverifying the received values.
Makefile: Reset environment for epydoc
epydoc allows overriding configuration values via environment variables.While this might be useful in certain use cases, but as no prefixwhatsoever is used, conflicts are easily created. Some people have theenvironment variable “NAME” set, effectively overriding the project name...
Add Ganeti.HTools.Graph
This module implements some algorithms on Data.Graph data structures.At the moment its main functionality is an LF-color implementation(greedy coloring in descending order of degree). There are also a fewextra functions to calculate the degree order, and convert the node to...
Small Makefile.am cleanups
Indentation within rules is fixed to always be 2 spaces (as opposed tomixed tabs/2 spaces/4 spaces). Additionally, the check-dirs rule,which was more complex than necessary, has been simplified: movedauto4mte.cache to DIRCHECK_EXCLUDE, and simplify the find call and the...
Fix long lines in Ganeti.HTools.Types and add Make check
Since hlint doesn't check style issues but rather code only issues,let's add a simple Makefile check for too-long lines, and fix thesingle bad case we have currently.
Signed-off-by: Iustin Pop <iustin@google.com>...
Two small haskell-related Makefile improvements
Just some cleanup:
- don't run hlint over hpc-htools, since it's just a symlink to htools.hs (so we'd get duplicate warnings)- build rpc-test in make really-all, via HS_ALL_PROGS variable
Improve the Haskell api doc directory creation system
Instead of having two different lists of directories, now there is only one.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Design for opportunistic locking
Add the design document for opportunistic locking.
Add unit tests for the JSON serialization of DRBD status
The serialization itself is done by Text.JSON, so the tests deal with checkingthat Text.JSON objects are created correctly from the DRBD parser datastructures.
Fix Haskell profiling targets
It turns out that in newer GHC versions, the suffix shouldn't beprefixed with a dot; otherwise name mangling is broken.
Fix Attoparsec test coverage check compatibility
The hpc tool, on squeeze, does not handle utf8 encoded files correctly.The Attoparsec test is just needed to verify whether the parser handlesuft8 correctly, so the solution is to exclude it from coverage checks....
ssconf: Verify file size when reading, add some tests
Until now ssconf would limit the amount read from files to 128 KiB andsilently ignored files larger than that. With this patch a check isadded by using fstat(2) on the file descriptor while it's being read....
Rename configure option for restricted commands
It is no longer “--enable-remote-commands”, but rather“--enable-restricted-commands”.
Include networks in Makefile and bash completion
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Iustin Pop <iustin@google.com>
Add DRBD parser unit tests
This adds tests that existing test files can be parsed by the Haskellparser as well, plus one new test file.
Add parser for DRBD /proc file
Split OpCode.hs and add module for opcode parameters
Due to TemplateHaskell stage restrictions, we can't define parametersin the same module as we're using them for TH, so we have to defineall module parameters in a separate module.
This patch therefore splits OpCodes.hs in two, adding that module and...
Create a new Ganeti/Types.hs module
There are already three cases where we copied type definitions betweenthe htools-specific types into the main ganeti code. Let's stop doingthis ☺ and create a common types module that holds these.
Note that there already exists BasicTypes.hs, but that refers to very...
configure: Shell cleanup
- Remote unnecessary quote characters- Change some uses of “test "$foo"” to “test n "$foo"” Merge “if/else/if” into “if/elif”- Wrap lines longer than 80 characters and add a test to “make check”
Prevent hlint crash on unicode chars
Some versions of hlint crash on the Attoparsec test file because theyare not able to correctly handle unicode characters if the proper encoding(--utf8) is not specified.
Rename design document for SSH setup
More design details will be added regarding setting up the node daemon,so this is more about adding a node than just setting up SSH.
backend: Implement remote commands
As per design document (doc/design-remote-commands.rst), a number ofrather strict tests is applied to any incoming request, a delay isinserted upon errors and returned error messages are very generic(unless it's the actual command that failed). There are unit tests for...
configure: Add option to enable remote commands
By default remote commands are disabled and need to be explicitelyenabled at build time.
Add ganeti-master-role.ocf example file
This allows controlling the cluster master role if the nodes are part ofa linux-HA cluster.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add ganeti-node-role ocf example file
This allows offlining nodes that don't respond if they are part of alinux-HA cluster.
Add new test for RAPI
Unlike existing tests, this actually tests RAPI at the interface withthe HTTP server. This way authentification can also be tested. A testfor “/2/query/…” is included as it's a bit special.
Automatically enable version-dependent GHC flags
Some GHC flags are very useful, but only appear in more recent GHCversions. To support the use of such flags while still supportingolder compilers, let's add conditional checks and enabling based onthe results....
Added attoparsec unit test for Unicode parsing
Attoparsec is known to have had issues with parsing non-ASCII strings.This test makes sure that parsing of Unicode characters works fine.
Signed-off-by: Michele Tartara <mtartara@google.com>[iustin: small doc string fixes]...
Improve make regen-vcs-version
"make regen-vcs-version" is run at every ./devel/upload time, in orderto have reasonably up-to-date information in the uploadedscripts/binaries. However, this means it currently triggersrecompilation of (at least) Ganeti/Version.hs, and sometimes more than...
Rename Confd.hs to Confd/Types.hs
This should be the last module rename, promise!
We rename this to conform to the other hierarchies (e.g. Query), andto not have both Confd.hs and Confd/*.hs.
Merge branch 'devel-2.6' into submit
Conflicts: Makefile.am (reordering, fixed) htools/Ganeti/Confd/Server.hs (hlint fixes on master) htools/Ganeti/Daemon.hs (hlint)...
Merge branch 'stable-2.6' into devel-2.6
Switch devel/upload to a static file
We had twice in the past days questions about devel/upload being"broken", since bash re-reads shell scripts during their run and thisfile can get regenerated due to Makefile changes.
Since we only need this to be dynamically built for 3 variables, let's...
Fix gnt-instance console with xl
- Rename xm-console-wrapper to xen-console-wrapper- Pass the xen command to use as a parameter
Remove setup-ssh
It has been superseeded by “prepare-node-join”.