qlang: Add parser for query filter language
With this parser, command line utilities will be able to provide filtersthrough query2 in a simplistic language. Example filters:
name "node3.example.com" master or (name "node4.example.com") be/memory == 128 and name =~ /^web/i...
htools: make some error messages more explicit
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add instance query field for OS parameters
These were not available as a query field before. Update unittestsand description text for the other “..params” fields.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
QA: also run gnt-cluster repair-disk-sizes
So that we don't happen again to break this forever without realisingit.
The patch also replaces one ' with ".
Fix shared_file_storage_dir on upgrades
If the cluster was upgraded from 2.4 or earlier, this key won't exist(it's only set to a correct value on cluster init), so we need toproperly set it to a null string (disabled).
Signed-off-by: Iustin Pop <iustin@google.com>...
QA: run the redist-conf command
This was (AFAICS) completely missing from the QA suite.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Prevent ssconf values from having non-string values
For whatever reason, my test cluster managed to acquireshared_file_storage_dir with a None value, instead of emptystring. This is not flagged in masterd itself, but the node daemonwill fail in writing the value to disk, as it calls len() on the...
Add some tests for the auto_balance attribute
It tests node add/remove secondary, rather than cluster-level N+1checks, but it's better than nothing.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Adeodato Simo <dato@google.com>
Node operations: take into account auto_balance
This patch changes the add to secondary/remove from secondary code tonot deduct/add the instance's memory if the instance is notauto_balanced.
Read/write auto_balance via Text
This also means another change in the text format; we really shouldmove to json…
The unittests are also update for the new 9-column layout andadditionally a bit of improvement is done.
Read auto_balance via Rapi
Read auto_balance via Luxi
Show the auto_balance flag in the instance listing
cli: Replace hardcoded strings with constants
Some more changes to Makefile.am for htools
I duplicate the BINARY= rule in the ghc invocation in order to be ableto silence the if, which was confusing.
Additionally, a new target for running just the htools unit-tests isprovided.
Add a new attribute to Instance.Instance
This will mirror Ganeti's be/auto_balance one, which we need to use toproperly match N+1 computations.
htools: Make opcode naming consistent with Ganeti codebase
This patch just cleans up the htools codebase to make it more consistentwith the naming of the Ganeti codebase.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge branch 'devel-2.4'
Merge branch 'stable-2.4' into devel-2.4
OpCodes.hs: make allow_failover optional
And default to False, like in the Python codebase.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
htools: add an utility function for JSON parsing
This allows extracting values from a JSON object that might miss, buthave a well-defined default value.
Two small Makefile fixes related to htools
First, fix hs-coverage on non-pristine tree, where the index.html filealready existed, and second, disallow compilation of htools binariesif configure, for some reason, didn't enable them.
htools: Use OpMigrateInstance with allow_failover option
Before hbal decided on the fly if an instance is migratable or not. Aswe implemented failover fallback in commit d5cafd31456 we can start touse that.
Signed-off-by: René Nussbaumer <rn@google.com>...
LUInstanceQueryData: Don't acquire locks unless requested
Until now LUInstanceQueryData always acquired locks for the instance(s)and nodes involved. In combination with long-running operations thisprevented the use of “gnt-instance info”, even with the “--static”...
gnt-instance migrate: Adding --allow-failover option
TLMigrateInstance: Merge failover code, allow fallback
As the code for failover for checking is almost identical it's an easytask to switch it over to the TLMigrateInstance. This allows us tofallback to failover if migrate fails prereq check for some reason....
Increase the lock timeouts before we block-acquire
This has been observed to cause problems on real clusters via thefollowing mechanism:
- a long job (e.g. a replace-disks) is keeping an exclusive lock on an instance- the watcher starts and submits its query instances opcode which...
utils: Add function generating regex for DNS name globbing
The intent of this function is to be able to provide a globbing operatoror query filters. One should be able to say, for example, something tothe effect of “gnt-instance shutdown '*.site'”.
Also rename a variable in MatchNameComponent....
Verify file consistency using centrally computed list
Until now “gnt-cluster verify” (LUClusterVerify) would compute its ownlist of files to check for consistency. This list was not complete andcertain inconsistencies were missed.
With this patch the code is changed to use the list of files used by...
cmdlib: Factorize computation of ancillary files
… and change the logic in _RedistributeAncillaryFiles. The virtuallysame list of files will be used to verify the files' consistency.
qlang: Remove OP_GLOB operator
It'll be implemented using OP_REGEXP by the parser.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
query: Add implementation of regex match operator
So far this operator was not implemented. This patch adds an additionalvalue preparation function to the function table for binary operators,used to compile the regular expression. Unittests are included....
cmdlib: Fix mistake made in commit 75c7520f0
Commit 75c7520f0 used the wrong constant. I double-checked all otherchanges made in the commit.
cmdlib: Replace hardcoded values with constants
daemon.py: move startup log message before prep_fn
Before this, the output in the rapi daemon log was:2011-04-04 03:09:51,026: ganeti-rapi pid=17447 INFO Reading users fileat /var/lib/ganeti/rapi/users2011-04-04 03:09:51,027: ganeti-rapi pid=17447 INFO ganeti-rapi daemon...
Display the actual memory values in N+1 failures
This changes the display from:Mon Apr 4 02:29:46 2011 * Verifying N+1 Memory redundancyMon Apr 4 02:29:46 2011 - ERROR: node node2: not enough memory toaccomodate instance failovers should node node1 fail...
Update iallocator.rst for multi-reloc mode
Signed-off-by: Adeodato Simo <dato@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
RAPI: Convert instance shutdown to the new FillOpCode
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix QA breakage caused by 3fd7f6524
ssh.VerifyNodeHostname: remove the quiet flag
This is not needed for this function, and can interfere with debuggingof ssh failures.
Add a simple wrapper over utils.Retry
The new wrapper makes moving legacy code to utils.Retry or addingretries in existing code simpler.
Expose whether htools was enabled to Python code
This exports whether htools was enabled at configure-time, and adds aconstant for our reference iallocator.
Automatically enable hail if enabled and found
test.ganeti.process_unittest: Fix race condition
There was a race condition on heavily loaded testsystem causing randomlyto fail the timeout unittests as the signal handler is not yet setup butthe timeout has already hit.
Therefore we introduce a workaround to wait until a program reached a...
Improve references to htools in the documentation
Was not sure about the bit in admin.rst, hope it's fine.
Clarify the need for QuickCheck/Haskell tests
Expands the devnotes.rst doc and adds warnings in the Makefile.
RAPI client: Remove support for version 0 instance creation requests
RAPI server: Drop support for instance creation format 0
Ganeti 2.1.3, released in June 2010, added support for a new, extensibleinstance creation request format, called version 1. This patch removessupport for the old and undocumented version 0 format....
Improved GanetiRapiClient docstrings
- Added @rtype and/or @return where missing- Fixed @param for Query() filter_ parameter (colon was missing)
Signed-off-by: Simeon Miteff <simeon.miteff@gmail.com>Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Add design for inter-group instance moves (multi-reloc)
iallocator.rst: refactor for readability; minor improvements
This commits breaks down the "Input message" section of iallocator.rst intotwo separate subsections: one detailing keys that are required in alloperation types; a second one detailing the "request" element, which is...
Add error checking and merging for cluster params
Set the default stderr logging level to WARNING so the relevant outputcan be seen.
Signed-off-by: Stephen Shirley <diamond@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Relax instance ERROR on admin_down on offline node
This fixes a issue, where an stopped instances is reported as ERRORin cluster verify if it lives on a offline node. As the instances isdown this shouldn't happen.
Implement submitting jobs from logical units
The design details can be seen in the design document(doc/design-lu-generated-jobs.rst).
iallocator.rst: give pointers for alloc_policy semantics
Doc fix in iallocator.rst: multi-evac requires "evac_nodes"
The request argument for multi-evacuate mode is "evac_nodes", not "nodes" (the example later in the file has the correct name already).
watcher: improve logging a bit
Add some debug logging to detail why we don't run some steps.
RAPI: Document need for Content-type header in requests
This was added to the NEWS file in commit ab221ddf, but neverdocumented properly.
Fix output for “gnt-job info”
If the result of an opcode was a non-empty dictionary, itwould be impossible to differenciate between input and result:
Input fields: […] debug_level: 0 fields: cluster_name,master_node,volume_group_name jobs: [[True, u'37922'], [True, u'37923'], [True, u'37924']]...
Remove old ensure-dirs (no longer needed)
Rewrite of ensure-dirs in python
I provided unittest to test the important pieces of the infrastructure.The one remaining function (ResuriveEnsure) is not easy to unittestbut also not critical if it fails to operate correctly.
hs-coverage: make a symlink hpc_index.html
This allows Apache to display the directory in a nicer way.
Another attempt at fixing htools build without curl
OK, my previous small fix was not good. There is another issue: haddoc(the documentation generator) needs to pass the same compiler options(i.e. in our case, -DNO_CURL) to ghc. But in case of no curl, then it...
Fix some lint warnings in htools code
hlint gives more suggestions, but some make the code (IMHO) harder toread.
Fix lint errors in the htools code
These are just changes from hlint suggestions. Still compiles andpasses unittests.
Add opcode summary to SubmitManyJobs errors
Requested-by: Iustin Pop <iustin@google.com>Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add design for submitting jobs from LUs
This patch adds a design document describing how jobs can be submittedfrom within LUs.
Fix Haskell unittests without RAPI
Since we don't test for now the RAPI backend directly, we can simplyskip the import. Later we can make a conditional import if needed.
Add import/export version 2 design document
Add design document for lighttpd as HTTP server
Add design document for X509 CA
Wrap long lines in configure.ac
- Use m4_normalize to make single-line strings while removing unnecessary spaces- Wrap lines longer than 80 characters
Update INSTALL and devnotes.rst with Haskell notes
This documents the needed libraries for Haskell development. It alsofixes a tiny typo in devnotes.rst.
Revert and change the apidoc and coverage dirs
Based on Michael's suggestion, this patch partially reverts mychanges. The new directories are:
- doc/api/py- doc/api/hs- doc/coverage/py- doc/coverage/hs
Basically the Python-specific output moves one level down (into py/)...
Rename away htools/NEWS
Also add mention about it being obsolete.
Cleanup the Haskell-related Makefile.am variables
This should be more readable now. I wanted to even use the nicer_SOURCES, but _SOURCES is special in Automake (again), so _SRCS it is.
Move hlint rule to the main Makefile
Since we do have errors currently, this is not enabled from the main'make lint' rule. That will get cleaned up later.
Remove obsolete htools/Makefile
Only one target wasn't ported over (the TAGS one), as hasktags is notavailable easily in distributions, so it doesn't make sense to enableit for all developers.
configure.ac: add checks for Haskell compiler/libs
This patch adds an enable/disable option for htools (--enable-htools),and associated tests for determining whether GHC (the compiler we usefor htools) and required libraries are present.
The method to do so is not very nice; usually, Haskell programs are...
Makefile.am: add rules for htools compilation
This patch adds some (not very nice) rules for htoolscompilation. Since automake only knows about some languages, anddoesn't have a real extension mechanism, I cheat and declare them as"buildable scripts", instead of programs. Rationale:...
Integrate htools's gitignore in top-level one
Remove some obsolete htools files
htools: install hail in the iallocator dir
Instead of bindir. We do this by adding a new allocdir variable, andredirecting hail to it.
htools: add option to control the use of RAPI
RAPI needs the curl module, which is a binding to the libcurl Clibrary. This is more complex (can be compiled either against gnutlsor openssl, etc.) and it's not of much help when the tools are used onthe cluster itself, we disable it by default....
Fix parallel compilation for htools
We do it via per-target-binary .o/.hi files. There are a few otheroptions, in the end this was chosen as different binaries can havedifferent options/components (e.g. hail won't use RAPI, etc.) and thatthe unittests need different compilation options....
Enable htools unittests and coverage reports
Since we have two sets of coverage data, we make the namingconsistent:
- doc/py-coverage for Python coverage- doc/hs-coverage for Haskell coverage
We also need to exclude some more dirs from the list of checked...
Enable htools apidoc generation and unify dir names
Previously, Python api doc was under doc/api (which didn't match withthe target rule, apidoc). After this patch, we have the following:
- make py-apidoc generates Python api doc under doc/py-apidoc- make hs-apidoc generates Haskell api doc under doc/hs-apidoc...
Port the live-test htools rule to the main Makefile
This was a bit tricky, as the compilation from the top-dir changes thepaths in the .tix/.mix files.
RAPI client: Tidy and test WaitForJobCompletion
- Use constants- Don't sleep if no delay is given- Mark function as deprecated: it uses polling instead of waiting for changes (but the latter needs authentication); it can still be used- Add unittests...
RAPI client: Add job status constants
RAPI client: Job IDs are strings
Split BuildHooksEnv of LUs
Commit dd7f677623 added another call to BuildHooksEnv to providepost-phase status variables. Since BuildHooksEnv also built the nodelists, that meant they have to be built twice. First a rather strictcheck was used, but it turned out to be more tricky. Commit b423c51336...
cfgupgrade: really ignore hostname when told so
cfgupgrade provides a --ignore-hostname option, however consequently itspawns config.ConfigWriter, which refuses to handle a foreignconfiguration file by default.
config.ConfigWriter provides a relative keyword argument,...
RAPI client: fix epydoc formatting
Add a helper function to the RAPI client
This adds a new method WaitForJobCompletion that can be used forclient who are not interested in the entire job log, just in itscompletion status.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>...
Move htools-ganeti 2.3 design doc to doc/
I didn't know where to put in the TOC, so I added it after the ganeti2.3 design doc.
Also, it seems that Sphinx is a bit limited in its list nesting(compared to Pandoc), so I had to rework a bit the indentation (it...
Integrate htools man pages in the main tree
This moves the man pages to their right place, removes the staticfooter and header, and adds Makefile rules for the new .1 man pages.
Remove restrictive hook node list check
Commit dd7f67762 added a restrictive check for the node lists returnedby BuildHooksEnv, leading to errors with some LUs, one of which wasfixed in commit 0dfa2c227. As it turns out, other LUs have similarissues, some not easy to fix. This patch disables the restrictive check...
watcher: Fix misleading usage output
When “ganeti-watcher” is called with an argument, it would hint ata non-existing “-f” parameter. With this patch the separate usagestring is no longer necessary.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Revert "Add temporary dir/file for htools import"
This reverts commit 8bc9e04a21457c4188475dc849d0d7d1e22ee929, notneeded anymore.