jqueue: Cache prepared field list in _JobChangesChecker
… instead of re-calculating it on every file change.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
NEWS: Deprecate LUXI calls replaced with query2
Adding the “luxi” namespace is necessary in “sphinx_ext”.
jqueue: Convert GetInfo to query2
This rather inefficient implementation (fields are evaluated on everycall to GetInfo) is not good for WaitForJobChanges and doesn't supportfilters, but that will be rectified in later patches.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
query: Add definitions for job fields
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
qlang.MakeFilter: Enable use of different name field
Jobs don't have a “name” field, so we must be able to controlthe field used for simple filters.
Merge cli.FormatTimestamp and utils.FormatTime
… to some degree at least. Unittests are included.
constants: Don't hardcode priorities for LOCK_ATTEMPTS_TIMEOUT
Also include unittest for LOCK_ATTEMPTS_TIMEOUT.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
jqueue._QueuedOpCode: Change a docstring
There was a typo and it's not necessary to repeat the class name.
locking: Remove unused OldStyleQueryLocks
No longer used after commit 090377807.
Update IP pool management design doc
Update IP pool managenent design doc to be consistentwith the implementation.
RPC: Add a new client type for DNS only
This patch moves the “call_version” to a new RPC client definition andthen adds a new runner using the DNS resolver for getting the hostaddress.
The standard “BootstrapRunner”, where the call was before, tries to...
LUNodeAdd: Make the version call only dependend on DNS
Also move the version check into prereq to abort before alter clusterstate if the version mismatch.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Update install document
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Update admin doc
Update walkthrough document
Update default instance kernel version
We switch from vmlinuz-2.6-… to vmlinuz-3-…. To do this nicely, wealso add a ./configure-time setting for the KVM instance kernel.
Update INSTALL and devnotes documents
Added the new Haskell library requirements, for both normal anddeveloper usage.
Furthermore, all commands are now converted to the shell-examplelexer.
Fix escaping of percent signs in the shell lexer
Of course, we do have cases where we want to escape the percent signs,and our regexes were not fully correct for this case.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Add a special lexer for sphinx/pygments
This will be used throughout our docs for better formatting exampleshell sessions, with custom markup for comments, user fixed input anduser variable input.
ganeti.7: Add more filter examples
Enable -Werror by default for htools
Since the code base is now "clean" across all supported GHC versions(6.12-7.4), we can enable -Werror again.
Switch to new-style exception handling
Currently, we're using Prelude.catch to handle I/O errors inhtools. This style of error handling has been deprecated for a while,but it still used to work without warnings.
However, the GHC release 7.4 has started to emit deprecation warnings...
Change a type computation for compatibility with 6.12
This is the last warning related to TemplateHaskell that was 6.12specific; for some reason, it doesn't "see" that traw/tname were used.
The patch just replaces the quoting syntax with an explicit type...
Fix compatibility with TemplateHaskell from GHC 7.4
GHC 7.4 has updated the TemplateHaskell library, and it turns out thatthe way we built the JSON instance implementation for showJSON was notgood (probably this is why GHC 6.12 was generating some warnings)....
Add trivial tests for gnt-* cli
While testing some other stuff, I realised that the gnt-* commandscould be broken (as in, the script fails with syntax errors), but makecheck doesn't detect it. Since we have shelltest, we can now addtrivial tests for this case....
Fix hardcoded Xen kernel path
We already have a ./configure-time variable for this, but it seems tobe actually unused.
Enable selection between Python and Haskell confd
This patch changes configure.ac Makefile.am so that the user can pass:
- --disable-confd (or --enable-confd=no) to disable it completely- --enable-confd=yes or --enable-confd=python to select the traditional implementation (this is the default setting)...
Fix qemu-img configure.ac check
By accident, commit a002ed7 introduced the qemu-img checks in thehtools block. I found this also by mistake while investigatinganother issue :)
Rework exit model
While updating the confd code, I realised that we have lots ofduplication in the exit model for the various programs.
So this patch attempts to abstract all the exits via a couple of newfunctions; sorry for the somewhat big patch, but I hope the payoff is...
Fix out-of-tree builds
The new shell tests do not succeed out-of-tree, due to static pathsand other issues. This trivial patch fixes these issue, make distchecknow passes.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Convert manual shell tests to shelltestrunner
This is more of a RFC… Basically most of the shell-based tests areconverted from exec+grep to shelltestrunner.
Things are not all fine and nice though:
- we have dependencies between tests, as some generate some data files...
Add command line option for controlling syslog use
… and enable it in hconfd.
Add support for syslog logging to Ganeti.Logging
Currently this is initialised to no from Daemon.hs, but will in thefuture allow command-line options for controlling it.
Only build hconfd if --enable-confd was passed
A later, more complete patch, will allow selecting between either thePython version or the Haskell version. This is just a temporarysolution to help building without all the needed Haskell libraries.
Signed-off-by: Iustin Pop <iustin@google.com>...
Build test helpers that point to hpc-htools
Instead of using just shell constructs to run hpc-htools correctly(i.e. HTOOLS=role htools/hpc-htools …), let's add some shell fragmentsthat do this for us.
This will ease the running of tests directly.
Allow hail to read data from stdin
This patch makes hail treat '-' as denoting stdin, per the usual Unixconvention. This will help with testing.
Update hconfd bind address handling
Instead of hardcoded IPv4 INADDR_ANY, this patch changes hconfd to useeither the any network for the configured cluster address family(ipv4/ipv6), or whatever the user passes in via the --bind option.
Add skeleton ssconf module
This currently has only one export function in it, which will be usedfor future bind address functionality in daemons.
Add two utility functions
These both are work with/on the Result type, so we add them toBasicTypes. The functions will be used as more generic versions ofsome more specialised functions that are right now spread across themodules.
Add the bind-address option
This implements the same logic as the Python code: if the option isnot used, use the default appropriate for the cluster, otherwise tryto parse and use whatever was passed in.
Typo fix: s/aditional/additional/
Trivial fix for a typo in message output of LUInstanceSetParams
Signed-off-by: Alexander Schreiber <als@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Fix exported constants
I "forgot" to run the unittests before commit :(
Export the AF_INET/AF_INET6 constants
These are needed to ensure that htools uses the same numeric values asPython.
By the way, I'm not sure what's the best option: importing socket intoconstants (this patch), or adding custom exports toautotools/convert-constants....
Rename htools-hpc to hpc-tools to fix laziness
This is a stupid/trivial patch. I didn't realise when addinghtools-hpc that this will "break" tab completion on htools; as in, Iwill have to add / manually always when completing files on thecommand line....
htools: fix long version of --port for daemons
The dashes do not need to be in, if they are then the resulting optionis:
-p PORT ----port=PORT Network port (default: 1814)
Add htools/rapi test directory to DIRS
Sorry!
Update gnt-node evacuate man page/help text
Fix gnt-group --help display
Copy-paste mismatch :)
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
hspace: Make use of the spindle_use
htools: Fix some compiler warnings
Add new hail (shell) tests
Using some hand-crafted files (manually modified from a real clusterrunning master branch), we test that hail behaves as we expect:doesn't load obvious wrong data, can do relocations/allocations, etc.
Enhance the RAPI backend to accept file:// URLs
This will allow offline testing of this backend (except for the actualcurl part), also in the case where we didn't compile against curl.
Add RAPI shell-level unittest
Enable pipefail mode for shell tests
These allow more accurate checks.
Add utilisation and instance selection tests
Add tests for custom text files
htools: Reflect the rename of spindle_usage to spindle_use
Rename spindle_usage to spindle_use on Ganeti side
Add a helper function for mirrorType computations
We most always use `templateMirrorType . diskTemplate`, so let's add ahelper function for this.
Fix TLMigrateInstance use of IAllocator
hail now expects correctly that relocate_from is of equal length withthe number of required nodes (fixme: there's a lot of not welldocumented behaviour here… not nice for any other potentialIAllocators). As such, we need to pass just the instance's primary...
Enable all disk templates in unit-tests
After the many patches implementing the shared storage functionalityin iallocator, balances, etc., and after the patches preparing theunit-tests to do the right thing for shared storage, we can finallyflip the bits that make instances have "random" disk templates as...
Enable evacuation/relocation for shared storage
This patch toggles the final bit and enables nodeEvacInstance to workon shared storage instances. Diskless instances are handled the same(as in, None is the same everywhere :).
Replace explicit case with 'either'
Instead of manually case-ing on the Either contents, let's just useeither with const functions (not really readable, but…).
Rename evacDrbdSecondaryInner to evacOneNodeInner
The function is now generic, so let's rename it and cleanup/improvethe docstrings.
Abstract the secondary evac into a separate function
This is generic enough (no longer depending on the replace type) thatit will be used for other replace modes too. So all we need to do isto abstract it to a separate function (that can be called from...
Fix Node_addPriFD test for arbitrary disk templates
Currently, this test assumes localStorageTemplates, so let's make surewe enforce that on the given instance
Extend the possible move types with FailoverToAny
This brings together all previous pieces and allows balancing to work(with the known caveats related to disk space calculation) for sharedstorage disk templates.
htools: Add the new ipolicy field
Also add the spindle usage to QC
ipolicy: Keep track of spindle usage
htools: Use the spindle usage in the algorithms
In the same transition also do a simple refactor of the caculation
htools: Convert all the other backends as this is trivial
htools: Adapt the Text backend to spindle usage
The old text files remain compatible with this change.
htools: Adding spindle usage to Instance
For now we just pass in "1" as spindle usage and adapt the backendlater.
iallocator: Add the spindle_usage to iallocator
This adapts the Ganeti side to export the spindle_usage
constants: Add new spindle_usage parameter to BE
Rework unit parsing
Due to how conversions were implemented previously, 1TB failed toparse on 32-bit, as we were overflowing during computation, eventhough the final result would fit easily.
This patch moves the parsing of the scaling factor to a separate...
Enable lvmstrap to run under Linux 3.x
Extend the kernel version check to also accept Linux 3.x as valid.
Signed-off-by: Alexander Schreiber <als@google.com>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix make distcheck after the new shell tests
With the new shell tests, we have for the first time the situationwhere tests require a binary built (via make), but the binary itselfis not the test. Hence, we need to declare and use check_SCRIPTS forit....
Build a coverage-enabled version of the tools
This allows the offline tests to also generate coverage data, and thehs-coverage target is changed to show both unit-test and offline-testscoverage.
The downside is that now we build yet-another-binary, which makes the...
Enhance the offline tests
This adds new offline tests:
- checks that files generated by hspace can be read by hbal/hinfo- checks that hbal handles node groups as expected- checks that enabling multiple backends results in failure- checks hspace machine readable code and higher verbosity...
Add more rebalance offline tests
These test that we can actually rebalance (and that it results inreplace-disks and failovers/migrations), and that we don'tdouble-rebalance, etc.
Enable htools offline tests as Haskell tests
This enables the offline-test.sh run both in the special "hs-check" rule and during normal "make check".
Add a simple binary tester for htools
This only tests the command line basic options. In the future, it canbe expanded to check other things.
Add a helper for htools CLI tests
… and update live-test.sh to use it, instead of hard-coding theprogram paths.
A few additional style fixes are also done (indentation).
htools: add confd server module
This contains a more-or-less complete implementation for theserver-side confd.
Note that most of the code is behaving identical to the Python code,with a notable exception: the asyncore/inotify code was changed (sinceHaskell doesn't have asyncore) to 3-thread system. While otherwise I...
htools: add confd daemon binary
This is currently not installed anywhere, just built.
htools: add basic daemon-related functionality
This is not complete for now, just the basic functionality has beenimplemented:
- daemonize- check we're running under the correct user- call setup logging
htools: add logging functionality
This adds a dependency on hslogger 1.1+, which is not available inDebian Squeeze, only in later versions.
htools: add runtime functionality
This duplicates the lib/runtime.py functionality, allowing us to checkfor runtime users and groups consistency.
htools: add a small module for crypto functions
This adds a dependency on the following new libraries:
- Data.HMAC module from the 'crypto' library- Data.Text from the 'text' library- Data.ByteString from the 'bytestring' library
In Debian, the bytestring library comes with the ghc compiler, whereas...
htools: add definitions for confd types
While we have some of these as plain types in Constants.hs, we addproper ADT definitions for them in a new file. Furthermore, we add theConfdRequest and ConfdReply types here (in Python they are inobjects.py).
htools: add a very basic lib/config.py equivalent
This only is able to load the configuration data for now.
htools: add partial implementation of lib/objects.py
This is partial since not all object types can be easily converted fornow (will need some changes on the Python side for this).
Most importantly, the *Params types do not have a good solution now:the Python code, due to its dynamic typing, hides the fact that we...
htools: Make it possible to load an IAlloc request
This is only useful in the stand-alone command line tools:
- hbal- hspace- hinfo
This change also documents the flag in all man-pages.
Fix lint error in commit 035b33e2
Commit 035b33e2 forgot one blank space, and current pylint in DebianSid doesn't run on our code… so I didn't realise this until aftercommit, sorry.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix relocation test ClusterAllocRelocate
As for the evacuate tests, we require a mirrored disk template, butotherwise the test should work for both mirror types.
Additionally, we perform a simplification that was left as TODO.
Fix IAlloc.processRelocate to support shared storage
Currently, this function expects (but doesn't really check! only viasecondary node being defined) a DRBD8 template type.
This patch (not very readable, sorry) changes it so that we checkwhether we're internally mirrored, externally mirrored, or not...
Fix up Cluster nodeEvac unittests for shared storage
Currently, these presume that the instance being generated always isDRBD/relocatable (but won't be in the future), and also presume thatall relocation modes can be used (hint: not ☺).
So we add a couple of helper functions to restrict the used disk...
Small improvement for nicer unit-testing
In case you just run "make htools/test" to recompile the unittests,the old .tix file should be removed, otherwise you get the messageabout inconsistent tick boxes. Let's make the compilation processautomatically remove this, for easier testing....
Rework evacDrbdSecondaryInner to take the imove
Instead of hard-coding the instance move operation, let's accept is asa parameter from the caller. This is the single thing that makes thisDRBD-specific, so removing it will help for futurepatches. Furthermore, this abstract the imove op type into a single...