Fix _checkRsaPrivateKey for newer key generation
Keys generated under debian sid just read "BEGIN PRIVATE KEY" ratherthan "BEGIN RSA PRIVATE KEY".
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix locking issues in LUClusterVerifyGroup
- Use functions in ConfigWriter instead of custom loops- Calculate nodes only once instances locks are acquired, removes one potential race condition- Don't retrieve lists of all node/instance information without locks...
cmdlib: Acquire BGL for LUClusterVerifyConfig
LUClusterVerifyConfig verifies a number of configuration settings. Fordoing so, it needs a consistent list of nodes, groups and instances. Sofar no locks were acquired at all (except for the BGL in shared mode)....
Export/import instance tags
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Fix issue with tags on instance creation
Commit 720f56c85a added the ability to specify tags when creating aninstance. The “tags” attribute of an instance object needs to be a set,but the patch's code saved it as a list, causing breakage in other parts...
Fix incomplete merge
Commit 66bd7445 changed the semantics of _JobProcessor on finishedjobs, and updated the related unittests in the 2.4 branch. It was thenmerged to master, however on master there was an additional test forthis case, which was not updated....
Export instance tags to instance hooks
Instance hooks now get an INSTANCE_TAGS environment variable, which contains aspace-delimited list of the affected instance's tags.
Also update the documentation to reflect the change.
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>...
Add tagging option to gnt-instance create
Add TAG_ADD_OPT option to cli.py and use it in gnt-instance. Modifycli.GenericInstanceCreate() accordingly.
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>...
Add tag handling to {Op,LU}InstanceCreate
Add a tag slot to opcodes.OpInstanceCreate. We do not reuse _PTags, as this isintended for OpTagsSet and thus:
a) is not documented b) does not carry a default value, making it mandatory
Also pass the tags to the iallocator during instance creation....
http.client: Make debug log less noisy
The HTTP client code generates quite a lot of debug log messages. Withthis patch they're hidden unless explicitely enabled in the code.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge branch 'devel-2.4'
htools: introduce a type alias for JSON objects
This makes the type definitions a bit more readable/simpler.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
hail: stop using old-style 'nodes' key
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
hail: add parsing of multi-relocate request
This is not handled yet, this patch just adds parsing of the incomingrequest.
hail: add option for displaying the parsed request
This can be used for debugging.
hail: add new data types for the multi-reloc mode
Add --no-instance-moves to the htools live tests
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Update hbal manpage for --no-instance-moves
Implement balancing with no instance moves
Note that --no-disk-moves and --no-instance-moves are not incompatible,but if both are used no solution can possibly exist.
Pass the instance moves option in hbal
While still being ignored, now it gets passed down to the iterationfunction.
Add --no-instance-moves cli htools option
This option doesn't currently do anything.
jqueue: Fix potential race condition when cancelling queued jobs
When a job was cancelled, its status would be changed and the filewritten again. Since this was a final status, the job file could bemoved anytime for archival. If the job was still in the queue, however,...
iallocator: add ht-checking for the request
Currently, we only ht-check the result value from the iallocator, andwe send whatever we happen to check manually in the LUs that call theiallocator.
This is not good, as we have to duplicate checks in many places, and...
iallocator: rename mem_size to memory
Currently, the iallocator in 'allocate' requires mem_size on inputbut serialises that as 'memory'. This inconsistency makes it hard toautomatically validate the parameters, hence this patch renamesmem_size.
Signed-off-by: Iustin Pop <iustin@google.com>...
iallocator: change default for target_groups
Per the design doc, the target_groups request key "if present, it musteither be the empty list, or contain a list of group UUIDs". Currentlyit defaults to None/null, which is not valid.
iallocator: export the hypervisor value
In 'allocate' mode, the documentation specifies that we export thehypervisor value (“Allocation needs, in addition: … hypervisor, thehypervisor of this instance”) and we need that on input, however wedon't actually export it....
iallocator: fix incomplete refactoring
Commit fdbe29ee changed the iallocator modes from 'r'/'w' to'ro'/'rw', but forgot one check in LUTestAllocator. This patch justcompletes the replacements.
gnt-node migrate: Use LU-generated jobs
Until now LUNodeMigrate used multiple tasklets to evacuate all primaryinstances on a node. In some cases it would acquire all node locks,which isn't good on big clusters. With upcoming improvements to the LUsfor instance failover and migration, switching to separate jobs looks...
Fix argument order in ReserveLV and ReserveMAC
ConfigWriter.ReserveLV() and Configwriter.ReserveMAC() calledTemporaryReservationManager.Reserve() with the ec_id and resource argumentsswapped. As a result, two reservation attempts for the same resource type...
ht: Accept both int and long as integers
This fixes a unittest failure on 32 bit systems. A recently addedunittest for ht.TJobId uses a rather large number (2347625220). On 64bit systems it is stored as “int”. On 32 bit systems however, Pythonuses “long”. The two types can be intermixed in Python as the...
Design doc for CPU pinning
Signed-off-by: Tsachy Shacham <tsachy@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
ht: Add checks for anything, regexp, job ID, container items
The check for container items is useful for tuples and/or lists withnon-uniform values. The “anything” check can be used when any valueshould be accepted for an item.
The job ID check, which uses the regexp check, will be used for...
Conflicts: lib/cmdlib.py: Trivial
TLReplaceDisks: Move assertion checking locks
Commit 1bee66f3 added assertions for ensuring only the necessary locksare kept while replacing disks. One of them makes sure locks have beenreleased during the operation. Unfortunately the commit added the check...
cli.JobExecutor: Handle empty name, allow adding job IDs
With LU-generated jobs only the ID is known.
cli.JobExecutor: Use counter for indexing jobs
If “SubmitPending” were mixed with calls to “QueueJob”, jobs in theinternal structures will get duplicate indices. With this change eachqueued job is assigned a unique index, which will be used for sorting...
Fix bug in LUNodeMigrate
Commit aac4511a added CheckArguments to LUNodeMigrate with a call to_CheckIAllocatorOrNode. When no default iallocator is defined,evacuating a node would always fail:
$ gnt-node migrate node123Migrate instance(s) '...'?y/[n]/?: y...
config: Add method to get members of nodes' groups
This will be used for locking during node evacuation.
Yet another attempt to fix builds
It seems that abs_top_srcdir is not a good option, so I tested againwith just using the same as in doc/examples/bash_completion.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Fix build breakage
Sorry, I already had PYTHONPATH exported in my env, and as I said Iwasn't able to test this on buildbot.
node evac: don't call IAllocator if no instances
Currently we generate an empty list only for the '-n node' invocation,but for iallocator we still call the iallocator (which needs an RPCcall, etc.). By moving the computation of instances outside of the if...
gnt-debug: rename allocator to iallocator
I'm always confused by this strange difference, so let's rename thecommand to match what it tests.
Misc other conversions
Convert job status strings to constants
Convert group policies to constants
Replace instance states hardcoded with constants
IAllocator.hs: replace a few strings with constants
Implement conversion of Python constants to Haskell
With the merge of the repositories, we can now auto-generate the codefor Haskell constants from the Python code.
Currently this only handles the basic types (strings andintegers). Handling containers such as lists and dictionaries is only...
RPC/Backend: Make UploadFile uid and gid agnostic
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Resolve uid/gid upon mainloop run
GetEntResolver: Make it possible to resolve uid/gid to name
utils.algo: Add InvertDict to invert a dict
autotools: Add noded group
Improve hooks documentation unittest
Also check for the opcode ID.
Update hooks.rst for cluster verify changes
Also update NEWS on this change.
Fix a couple of style mistakes
Cluster verify: accept a --node-group option
This will trigger a ClusterVerifyGroup operation only on the specifiedgroup, skipping other groups as well as cluster-wide verifications.
Signed-off-by: Adeodato Simo <dato@google.com>Signed-off-by: Guido Trotter <ultrotter@google.com>...
Cluster verify: check for nodes/instances with no group
Previously, all nodes and instances would always be visited/verified. Bydriving the verification by node group now, we will miss nodes andinstances that can't be reached from existing node groups, should that rare...
Cluster verify: fix LV checks for split instances
When sharding by group, if a mirrored instance is split (primary andsecondary) between two groups, its volumes will not be properly checked:the group of the primary will warn about a missing volume in the secondary,...
Cluster verify: make NV_NODELIST smaller
To cope with increasing cluster sizes, we now make nodes try to contact allother nodes in their group, and one node from every other group.
Cluster verify: verify hypervisor parameters only once
The list of all hypervisor parameters has to be computed inLUClusterVerifyGroup, since it needs to be passed to nodes asNV_HVPARAMS. However, it is better only to verify said parameters once,out of LUClusterVerifyConfig....
Split LUClusterVerify into LUClusterVerify{Config,Group}
With this change, LUClusterVerifyConfig becomes a "light" LU that onlyverifies the global config and other, master-only settings, and the bulk ofnode/instance verification is done by LUClusterVerifyGroup, which only acts...
Cluster verify: factor out error codes and functions
We move all error code definitions, plus the _Error and _ErrorIf helpers,to a private _VerifyErrors mix-in class that can be later shared by the newtwo cluster verify LUs.
(_Error and _ErrorIf code was moved around verbatim, except to disable...
Cluster verify: make "instance runs in wrong node" node-driven
Previously, the "instance should not be running in this node" error wascomputed by verifying, for each instance, whether any node other than itsprimary was running it. But this is not a well-suited approach if we were...
Verify an absent vm_capable node for files
If we're not verifying all nodes, adding a node outside the currentgroup for file checksums helps us making sure checksums are the same inall of the cluster.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Cluster verify: master must be present for _VerifyFiles
This commit prepares the call to _VerifyFiles for the case when the masternode is not one of the nodes that's being verified (which will be the casefor all node groups but one). We fix it by always passing master info and...
Cluster verify: don't assume we're verifying all nodes/instances
This commit fixes a few initial simple cases in which it was assumed thatwe're always working over the whole cluster. With this change, wedifferentiate between "nodes/instances to verify" and "checks that need...
Cluster verify: gather node/instance list in CheckPrereq
This commit introduces no behavior changes, and is only a minor refactoringthat aids with a cleaner division of future LUClusterVerify work. Thechange consists in:
- substitute the {node,instance}{list,info} structures previously created...
Merge remote branch 'origin/devel-2.4'
cli: Replace hardcoded disk templates with constants
mcpu: Add missing docstring to _ProcessResult
config: Add function to get instances in node group
This will be used for evacuating instances in a node group.
iallocator: Stricter check for multi-evac result
Check new secondary nodes' group like it's already done formulti-relocation requests.
cmdlib: Use ganeti.ht for checking iallocator result
Fix errors in hooks documentation
In many cases the opcode ID was incorrect. A unittest for this willbe added in the master branch.
ht: Add strict check for dictionaries
This allows checking specific dictionary items, unlike TDictor TDictOf.
cmdlib: Remove punctuation from error messages
Various grammar fixes and updates
Signed-off-by: Stephen Shirley <diamond@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
gnt-debug: New iallocator mode
Add new iallocator mode to LUTestAllocator
cmdlib.IAllocator: Add multi-relocate support
Add constants for multi-relocation iallocator mode
Clarify a bit the noded man page
"This can be overriden" can be read as either the port we listen on orthe address we bind to. Replace with "The port" for great clarity!
Note --no-remember in NEWS
Switch QA over to using instance stop --no-remember
Instead of hardcoded Xen commands. This will make it work for allhypervisors, instead of duplicating hypervisor functionality in QAitself.
The timeout has been removed as gnt-instance stop itself will make...
Implement no_remember at RAPI level
Implement no_remember at CLI level
Introduce instance start/stop no_remember attribute
This will allow stopping or starting an instance without changing theremembered state. While this seems counter-intuitive at first (it willcreate cluster verify errors), it can help in a few corner cases:...
cmdlib.IAllocator: Fewer temporary variables
Reduce the number of temporary variables and generate dictionaries inone go.
TLMigrateInstance: do not migrate to self
Check that the instance is not being migrated to its current primary nodeduring CheckPrereq. Otherwise migration is aborted because the instance isalready running and cleaned-up, which causes the running instance to be killed....
SharedLock: Implement downgrade from exclusive to shared mode
If a job needs to modify a resource and then wait for a result, it mustacquire the resource lock in exclusive mode. In some cases it would bepossible to only have a shared lock for waiting. Until now it was not...
gnt-debug: Use constants for iallocator direction
Use disk mode constants in iallocator documentation
gnt-debug, opcodes: Use constants for iallocator
cmdlib.IAllocator: Use lookup table for mode-specific data
Bump version for the 2.4.2 release
I think we should stop finding bugs and instead release this :)
Abstract ignore_consistency opcode parameter
Two opcodes already use it and we need it for a third, time to add aconstant for it.
Fix a bug in LUInstanceMove
The opcode parameter ignore_consistency was used in the LU, but notactually declared in the OpCode. The patch adds it in the opcode andthe command line client.
ObQuote — Please, please, can I have static typing?