RAPI: Convert instance shutdown to the new FillOpCode
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix QA breakage caused by 3fd7f6524
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Add a simple wrapper over utils.Retry
The new wrapper makes moving legacy code to utils.Retry or addingretries in existing code simpler.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Automatically enable hail if enabled and found
Expose whether htools was enabled to Python code
This exports whether htools was enabled at configure-time, and adds aconstant for our reference iallocator.
test.ganeti.process_unittest: Fix race condition
There was a race condition on heavily loaded testsystem causing randomlyto fail the timeout unittests as the signal handler is not yet setup butthe timeout has already hit.
Therefore we introduce a workaround to wait until a program reached a...
Improve references to htools in the documentation
Was not sure about the bit in admin.rst, hope it's fine.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Clarify the need for QuickCheck/Haskell tests
Expands the devnotes.rst doc and adds warnings in the Makefile.
RAPI client: Remove support for version 0 instance creation requests
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
RAPI server: Drop support for instance creation format 0
Ganeti 2.1.3, released in June 2010, added support for a new, extensibleinstance creation request format, called version 1. This patch removessupport for the old and undocumented version 0 format....
Improved GanetiRapiClient docstrings
- Added @rtype and/or @return where missing- Fixed @param for Query() filter_ parameter (colon was missing)
Signed-off-by: Simeon Miteff <simeon.miteff@gmail.com>Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Add design for inter-group instance moves (multi-reloc)
Signed-off-by: Adeodato Simo <dato@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
iallocator.rst: refactor for readability; minor improvements
This commits breaks down the "Input message" section of iallocator.rst intotwo separate subsections: one detailing keys that are required in alloperation types; a second one detailing the "request" element, which is...
Relax instance ERROR on admin_down on offline node
This fixes a issue, where an stopped instances is reported as ERRORin cluster verify if it lives on a offline node. As the instances isdown this shouldn't happen.
Signed-off-by: René Nussbaumer <rn@google.com>...
Implement submitting jobs from logical units
The design details can be seen in the design document(doc/design-lu-generated-jobs.rst).
iallocator.rst: give pointers for alloc_policy semantics
Doc fix in iallocator.rst: multi-evac requires "evac_nodes"
The request argument for multi-evacuate mode is "evac_nodes", not "nodes" (the example later in the file has the correct name already).
watcher: improve logging a bit
Add some debug logging to detail why we don't run some steps.
Remove old ensure-dirs (no longer needed)
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Rewrite of ensure-dirs in python
I provided unittest to test the important pieces of the infrastructure.The one remaining function (ResuriveEnsure) is not easy to unittestbut also not critical if it fails to operate correctly.
hs-coverage: make a symlink hpc_index.html
This allows Apache to display the directory in a nicer way.
Another attempt at fixing htools build without curl
OK, my previous small fix was not good. There is another issue: haddoc(the documentation generator) needs to pass the same compiler options(i.e. in our case, -DNO_CURL) to ghc. But in case of no curl, then it...
Fix some lint warnings in htools code
hlint gives more suggestions, but some make the code (IMHO) harder toread.
Fix lint errors in the htools code
These are just changes from hlint suggestions. Still compiles andpasses unittests.
Add opcode summary to SubmitManyJobs errors
Requested-by: Iustin Pop <iustin@google.com>Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add design for submitting jobs from LUs
This patch adds a design document describing how jobs can be submittedfrom within LUs.
Fix Haskell unittests without RAPI
Since we don't test for now the RAPI backend directly, we can simplyskip the import. Later we can make a conditional import if needed.
Add import/export version 2 design document
Add design document for lighttpd as HTTP server
Add design document for X509 CA
Wrap long lines in configure.ac
- Use m4_normalize to make single-line strings while removing unnecessary spaces- Wrap lines longer than 80 characters
Update INSTALL and devnotes.rst with Haskell notes
This documents the needed libraries for Haskell development. It alsofixes a tiny typo in devnotes.rst.
Revert and change the apidoc and coverage dirs
Based on Michael's suggestion, this patch partially reverts mychanges. The new directories are:
- doc/api/py- doc/api/hs- doc/coverage/py- doc/coverage/hs
Basically the Python-specific output moves one level down (into py/)...
Rename away htools/NEWS
Also add mention about it being obsolete.
Cleanup the Haskell-related Makefile.am variables
This should be more readable now. I wanted to even use the nicer_SOURCES, but _SOURCES is special in Automake (again), so _SRCS it is.
Remove obsolete htools/Makefile
Only one target wasn't ported over (the TAGS one), as hasktags is notavailable easily in distributions, so it doesn't make sense to enableit for all developers.
Move hlint rule to the main Makefile
Since we do have errors currently, this is not enabled from the main'make lint' rule. That will get cleaned up later.
Port the live-test htools rule to the main Makefile
This was a bit tricky, as the compilation from the top-dir changes thepaths in the .tix/.mix files.
Enable htools apidoc generation and unify dir names
Previously, Python api doc was under doc/api (which didn't match withthe target rule, apidoc). After this patch, we have the following:
- make py-apidoc generates Python api doc under doc/py-apidoc- make hs-apidoc generates Haskell api doc under doc/hs-apidoc...
Enable htools unittests and coverage reports
Since we have two sets of coverage data, we make the namingconsistent:
- doc/py-coverage for Python coverage- doc/hs-coverage for Haskell coverage
We also need to exclude some more dirs from the list of checked...
Fix parallel compilation for htools
We do it via per-target-binary .o/.hi files. There are a few otheroptions, in the end this was chosen as different binaries can havedifferent options/components (e.g. hail won't use RAPI, etc.) and thatthe unittests need different compilation options....
htools: add option to control the use of RAPI
RAPI needs the curl module, which is a binding to the libcurl Clibrary. This is more complex (can be compiled either against gnutlsor openssl, etc.) and it's not of much help when the tools are used onthe cluster itself, we disable it by default....
htools: install hail in the iallocator dir
Instead of bindir. We do this by adding a new allocdir variable, andredirecting hail to it.
Remove some obsolete htools files
Integrate htools's gitignore in top-level one
Makefile.am: add rules for htools compilation
This patch adds some (not very nice) rules for htoolscompilation. Since automake only knows about some languages, anddoesn't have a real extension mechanism, I cheat and declare them as"buildable scripts", instead of programs. Rationale:...
configure.ac: add checks for Haskell compiler/libs
This patch adds an enable/disable option for htools (--enable-htools),and associated tests for determining whether GHC (the compiler we usefor htools) and required libraries are present.
The method to do so is not very nice; usually, Haskell programs are...
RAPI client: Tidy and test WaitForJobCompletion
- Use constants- Don't sleep if no delay is given- Mark function as deprecated: it uses polling instead of waiting for changes (but the latter needs authentication); it can still be used- Add unittests...
RAPI client: Add job status constants
RAPI client: Job IDs are strings
Split BuildHooksEnv of LUs
Commit dd7f677623 added another call to BuildHooksEnv to providepost-phase status variables. Since BuildHooksEnv also built the nodelists, that meant they have to be built twice. First a rather strictcheck was used, but it turned out to be more tricky. Commit b423c51336...
cfgupgrade: really ignore hostname when told so
cfgupgrade provides a --ignore-hostname option, however consequently itspawns config.ConfigWriter, which refuses to handle a foreignconfiguration file by default.
config.ConfigWriter provides a relative keyword argument,...
RAPI client: fix epydoc formatting
Add a helper function to the RAPI client
This adds a new method WaitForJobCompletion that can be used forclient who are not interested in the entire job log, just in itscompletion status.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>...
Move htools-ganeti 2.3 design doc to doc/
I didn't know where to put in the TOC, so I added it after the ganeti2.3 design doc.
Also, it seems that Sphinx is a bit limited in its list nesting(compared to Pandoc), so I had to rework a bit the indentation (it...
Integrate htools man pages in the main tree
This moves the man pages to their right place, removes the staticfooter and header, and adds Makefile rules for the new .1 man pages.
Remove restrictive hook node list check
Commit dd7f67762 added a restrictive check for the node lists returnedby BuildHooksEnv, leading to errors with some LUs, one of which wasfixed in commit 0dfa2c227. As it turns out, other LUs have similarissues, some not easy to fix. This patch disables the restrictive check...
Revert "Add temporary dir/file for htools import"
This reverts commit 8bc9e04a21457c4188475dc849d0d7d1e22ee929, notneeded anymore.
Merge remote branch 'htools/master'
Add temporary dir/file for htools import
In order for “git merge -s recursive -X subtree=htools” to do itsmagic, an (mostly empty) directory must exist in the target tree name‘htools’. Hence we add a temp file to it before the merge, and removeit after that....
Fix hook node list when adding node
This broke QA (and everyone trying to add a node) by complaining aboutdifferent node lists.
hooks: Provide variables with post-opcode values
When a hook is called, it is provided with a number of variablesdescribing the status of the instance/node/etc. before the operation.Some opcodes provide extra variables to see modified values from hooks,...
HooksMaster: Add more assertions for variable names
Also replace explicit loop with dict.update.
mcpu: Tidy HooksMaster a bit
- Dictionary indentation- Add empty lines for readability- Simplify conditional code
cmdlib: Factorize running post-pase hook
QA: Add tests for queries via RAPI
qlang: Remove unused import
RAPI: Add support for querying resources
- Access is only permitted for authenticated clients (queries can return sensitive data)- Filters can be specified when sending a PUT request- Updates RAPI client, documentation and tests
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Add support for query resources in RAPI URIs
masterd: Simplify code for field queries
Instead of going via cmdlib and using special cases for differentresources, the list of fields is used directly.
constants: Rename QR_OP_*, add QR_VIA_RAPI
Commit 28b71a76 added a list of resources which can be queried usingLUXI. Unfortunately the variable was named “QR_OP_LUXI”, which can beconfusing. This patch renames “QR_OP_QUERY” to “QR_VIA_OP”, “QR_OP_LUXI”...
qlang: Remove unused ReadSimpleFilter
TLReplaceDisks: Add check if disks are activated
Previously we failed later with a rather useless error message. Thispatch fixes this and tells the user to activate-disks if replace-disksis in the need of activated disks rather than abort with a cryptic error...
LUOsDiagnose: Move legacy behaviour into filter
The behaviour of LUOsDiagnose needs special treatment. Commit d22dfef7changed it to not return hidden, blacklisted or invalid OSes if therespective field is not requested. This behaviour needs to be preserved...
Convert OsDiagnose to query
qlang: Add some more documentation for filters
It's not perfect, but at least some more.
query: Add conversion wrapper
Allows converting the value of a column before returning it. Useful forsorting while still using one of the other generic functions forretrieving the value.
Fix epydoc warning about unknown reference
config: Wrap MatchNameComponent, reduce lock duration
- Remove duplication by merging two MatchNameComponent into a wrapper- Reduce lock duration by getting list of names under lock and then matching names without the lock- Also, ExpandNodeName's docstring is fixed....
opcodes: Document OpQueryFields' parameters
Update NEWS file for version 0.3.1
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Update copyright years in the man pages
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>
Remove obsolete env vars from the manpages
Commit 16c2369c removed support for using environment variables forselecting the text backend (since now it's just one option), butdidn't update the man pages.
Instance failover: fix bug for INT_MIRROR cases
Patches db366d9a and aac4511a added support for EXT_MIRROR instances,but inadvertently introduced a bug: for INT_MIRROR cases, we don'tneed (actually we can't support) neither an iallocator nor a targetnode....
gnt-cluster epo: Adding --power-delay flag
gnt-node power: Adding --power-delay flag
cli.py: Adding POWER_DELAY_OPT
The command line option --power-delay sets the time waited between powerons.
OpOobCommand: Adding power on delay
This delays the invocation of the power on of the next node. So if youpower on a bunch of nodes it will not blow the fuse.
OpOobCommand: Document all fields
gnt-cluster epo: Adding --shutdown-timeout
This adds the --shutdown-timeout flag to gnt-cluster epo to specify theshutdown timeout for instance shutdown.
Rename DTS_NET_MIRROR to DTS_INT_MIRROR
DTS_INT_MIRROR better contrasts DTS_EXT_MIRROR.
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>[iustin@google.com: updated patch for changed context]Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Shared file support for tools/burnin
Make burnin work with -t sharedfile and update burnin's help.
Shared block storage support is not included, as currently there is no way toprovision shared block storage from within Ganeti.
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>...
KVM: use cache=none for shared disk templates
Disable host cache for externally mirrored disks to avoid cache incoherency.Without this, migrations between the same two nodes may end up in diskcorruption.
This is a runtime override of cluster defaults, mostly a workaround....
Shared storage instance failover
Modify LUFailoverInstance to enable shared storage instances to failover.Shared storage instance failover requires either a target node or aniallocator to determine the target node. If none is given, the cluster default...
Shared storage node migration
Modify LUNodeMigrate to provide node migration for nodes with instances usingshared storage. gnt-node migrate has to be passed an iallocator for migrationof shared storage instances to be performed. When using a shared storage...
Shared storage instance migration
Modify LUMigrateInstance and TLMigrateInstance to allow instance migrations forinstances with DTS_EXT_MIRROR disk templates.
Migrations of shared storage instances require either a target node, or aniallocator to determine the target node. If none is given, the cluster default...
CLI changes to facilitate shared storage migration/failover
Add DST_NODE_OPT to cli.py to use for directly specifying the target nodeduring migration/failover.
gnt-instance failover/migrate also get passed an iallocator option.
gnt-node failover/migrate get only a target_node option....
Migration and failover: add iallocator and target_node slots
Add iallocator and target_node slots to OpMigrateInstance andOpFailoverInstance to facilitate shared-storage-backed instance mobility. Addiallocator slot to OpMigrateNode (no explicit target_node in this case)....
IAllocator changes to work with shared storage
Make cmdlib.IAllocator shared-storage-aware. IAllocator requires secondarynodes only on DTS_NET_MIRROR disk templates and requires no secondaries forDTS_EXT_MIRROR templates.
Shared block storage support
This patch introduces basic shared block storage support.
It introduces a new storage backend, bdev.PersistentBlockDevice, touse as a backend for shared block storage. The new bdev requires a newBLOCKDEV_DRIVER_MANUAL constant with the value "manual" and uses it as...