htools: replace two hardcoded uses of pri+sec nodes
These two cases use explicit uses of primary and secondary nodes withInstance.allNodes, which means the code is more flexible if theinternal layout of the instance changes.
I've verified that the output of involvedNodes is not required to be...
htools: add target_node member to migrate opcode
… and failover too. Not many changes otherwise except forserialisation and unittests.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
htools: do not change node disk for non-local storage
htools: add more functions for local disk storage
These will be used in Node.hs for proper add/remove instancecode. Furthermore, we restrict the movable status to the right disktemplates only, so that we don't attempt to move the 'wrong' instancetypes....
Initial design doc for OVF support
Signed-off-by: Agata Murawska <agatamurawska@google.com>[iustin@google.com: fixed formatting issues]
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
gnt-instance console: Use query instead of opcode
This means opening the console no longer requires the instance lock,allowing it to be used during long-running operations (e.g. replacing adisk).
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge branch 'devel-2.4'
Add opcode attribute for comments
This attribute allows programmatic submitters of jobs (e.g. iallocator)to add a comment to each opcode, describing its purpose. Example:
$ gnt-job info 123Job ID: 123 … Opcodes: OP_INSTANCE_REPLACE_DISKS …...
gnt-node volumes: Fix instance names
Commit 84d7e26b changed “objects.Instance.MapLVsByN” to not just returnthe LV name, but to include the volume group name (e.g.“xenvg/d67e8700….disk0_data”). This in turn broke the mapping of volumenames in LUNodeQueryvols, stopping instance names from displayed in...
Fixed one option name and a typo in the docs
The -g vg-name option was deprecated in commit04367e70ad71eea3f0f19e7889dc68fb9783c98a.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix instance failover (missing argument)
More fallout from commit 323f9095b49d.
Implement instance failover via RAPI
No idea why this was missed before.
Make lock monitor more versatile
With this change it'll be possible to register other lock informationproviders. One usecase for this are job dependencies, which can be shownin the output of “gnt-debug locks”, too.
The lock monitor is changed to accept more than one return value from...
locking.GLM: Allow adding locks to monitor
This will be used for exporting job dependencies throughthe lock monitor.
Export job dependencies through lock monitor
This makes them visible to the user. Example:
$ gnt-debug locks -o name,pendingName Pendingjob/890 job:891,892job/892 job:894
Update documentation regarding Haskell dependencies
These were forgot when the supported library versions were changed.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
htools: add two more small unittests
This adds tests for the opToResult and eitherToResult functions fromTypes.hs, and changes two other tests for the same module to test JSONserialisation (which automatically also tests the lower-level to/fromstring conversion functions)....
htools: update hail man page with the new modes
Also mark the deprecated modes we no longer support.
htools: a few more hlint fixes
Tested only on GHC 7.x, will test on 6.1x too before commit.
htools: further docstring fixes
This adds parameter documentation for Cluster.iMoveToJob (I think itwas not clear if the new or old node list is needed) and fixes otherdocstring style issues.
After this patch, all modules except for CLI.hs (which has many...
htools: add JSON instance for EvacMode
This abstracts the JSON parsing of the type EvacMode near itsdefinition, and simplifies its conversion in IAlloc.parseData.
htools: add human-readable output to hspace
Currently, hspace can only output a machine-readable format that(while detailed) is hard to parse quickly by people. This patch adds(and enables by default) a human-readable output that shows the mostimportant metrics in a simple format....
Fix job constants use in htools
Commit 56c094b4 added use of job constants, but I didn't payattention and ended up mixing things: job constants were used foropcode ones, and the job ones didn't get converted.
This patch corrects it and uses only C.* constants throughout the Jobs...
Add error state to LUGroupEvacuate's exceptions
Rename *_STATUS_WAITLOCK to …_WAITING
This patch renames the {JOB,OP}_STATUS_WAITLOCK constants to {JOB,OP}_STATUS_WAITING, as per design document for chained jobs.
gnt-group: Add command to evacuate whole group
Add new opcode for evacuating group
Fix locking issue with job dependencies
When jobs waiting for a dependency are notified, they're re-added to thequeue. This would require owning the queue lock in exclusive mode, butsince the function doing so is called from within the job/opcodeprocessor, it only holds the lock in shared mode....
jqueue: Read-only jobs don't need processor lock
Add support for KVM keymaps
Signed-off-by: Sébastien Bocahu <zecrazytux@zecrazytux.net>Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
gnt-debug: Add tests for job dependencies
jqueue: Implement submitting multiple jobs with dependencies
With this change users of the “SubmitManyJobs” interface can userelative job dependencies. Relative job IDs in dependencies are resolvedbefore handing the job off to the workerpool.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Fix node evacuation
- Adjust for new iallocator result format- Split some code into helper functions
Do proper name lookup for the -O option
hspace and hbal treat -O differently, and use aliases for short names(although hbal succeeds in that, and hspace doesn't). Uniform this witha name lookup, using the same functions we used for instanceselection/exclusion....
jqueue: Add “writable” flag to memory objects
Basically only one instance of the job, the one being processed,should be serialized to disk and replicated to other nodes. Withthis flag assertions can be added in various places.
Implement chained jobs
An overview is available in the design document for this change,doc/design-chained-jobs.rst.
When a job enters the job processor, the current opcode's dependenciesare evaluated. If a referenced job has not yet reached the desired...
Add implementation details to design for chained jobs
As requested by Iustin.
Add support for GPT by using parted for disks bigger than 2TB.
Signed-off-by: Pedro Macedo <pmacedo@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Remove constants for iallocator multi-relocate
They're no longer necessary.
htools: add a machine-readable CLI flag
This will be used in hspace to toggle between "human" readableand machine readable output formats.
htools: move the '-p' option to htools.rst
Since this is a common option and has a big description.
htools: switch hspace to the generic binary
This is the last patch of the binaries conversion.
As information, we now have a single binary that is approx. 5.4MiB insize, compared to 4 binaries that were approx. 5.1-5.2MiB in size;this will result in a smaller package and install size, and the single...
htools: import the program modules in QC.hs
This adds the binaries code to the coverage, and thus the coveragefinally shows the real coverage over all logic code (except for thehtools.hs code, which is not logic code related to the algorithms, soit doesn't matter — plus it's also very small)....
htools: move tiered spec map helper to Hspace.hs
This is used just in hspace, so let's help in making Cluster.hssmaller. We also split the function in two, as computing the spec mapand formatting it are two different tasks.
Signed-off-by: Iustin Pop <iustin@google.com>...
htools: switch hscan to the generic binary
htools: switch hbal to the generic binary
In addition, the patch adds a separate Makefile variable for holdingthe binary roles to make it more clear what we symlink.
htools: switch hail to the generic binary
This converts the first binary to the generic 'htools' binary.
htools: add a generic binary
This is the start of a series of patches that will unify all thebinaries currently in use in a single one, which can perform differentroles based on the name it is installed as.
htools: add a compatibility module
When compiling with the parallel-3.x library, we get a deprecationwarning, which makes understanding any other error messagesharder. This patch adds a compatibility module that will hold suchcode for transitioning libraries....
htools: remove no-longer-needed tryMG* functionality
… which was deprecated by the previous patch.
htools: remove ialloc/relocate and multi-evacuate
Since the new node-evacuate mode does both their work and alsosupports better multi-group clusters (including handling splitinstances).
htools: fix potential bug in ialloc/change-group
Currently, the ChangeAll mode of nodeEvac computes the primary groupof the instance and then uses the resulting group index for computingthe group score. However, during the change-group operation (which...
htools: run IAllocator input through checkData
As the IAllocator backend is using a different data path than theothers, it doesn't get the full functionality that loadExternalDatadoes. This results in the current situation where checkData is not runon the input cluster state, which means the node memory properties are...
htools: abstract a function for displaying warnings
This will make it possible to reuse this in IAllocator too.
htools: use maybePrintNodes in hail.hs
This eliminates duplication of codes (and was forgotten back whenmaybePrintNodes was added).
htools: add cluster state saving support to hail
This add support for saving the cluster state (both pre- andpost-iallocator run) to a text file such that it can be fed back intoany of the htools commands.
htools: return the final instance map in ialloc
Similar to the previous patch, this returns the final instance mapfrom the iallocator run, which will allow saving the cluster state forfurther examination/post-processing.
htools: implement post-alloc cluster status display
This patch changes the IAllocator result formatting workflow to returnthe final node list, which can be then used to display the final nodestatus too—currently only the initial status can be shown, which is...
Update node group iallocator design to use job dependencies
While working on a function to submit jobsets, I realized that weactually don't need them anymore. With the new job dependencies, theiallocator plugin can just generate the right dependencies and gets the...
Fix assertion error on unclean master shutdown
Commit 66bd7445 added an assertion to ensure a finalized job has its“end_timestamp” attribute set. Unfortunately it didn't cover a case whenthe queue is recovering from an unclean master shutdown.
Make SharedLock._is_owned public
This will be useful for assertions. GanetiLockManager._is_owned isexported, too.
htools: return new state from new IAllocator modes
The old modes already return the node list (as part of AllocSolution),this patch makes the new modes provide this new information.
htools: update the node-evac/change-group result
… per the new design changes. This also requires changes toupdateEvacSolution, as passing in the original instance is nothelpful; we just pass the instance index now and look up the actualinstance in the modified instance map....
htools: add function for getting all nodes of an instance
htools: add change-group IAllocator mode
This builds a lot upon the node-evacuation mode, and itsimplementation is therefore quite easy (once node-evac was added).
htools: add group restrictions to findBestAllocGroup
While a normal multi-group allocation doesn't need to restrict thelist of target groups beyond the unallocable status of some groups,when we relocate instances from one group to the other we need tolimit the subset of searched groups....
htools: generalise and rename availableLocalNodes
Currently this function works in the context of an instance, computingits local group nodes. We can use generalise it to take a group indexargument, so that it's not bound to an instance, and we rename it...
htools: add ChangeGroup to IAllocator types/loader
This patch updates the types and functions in IAllocator.hs andLoader.hs to the new design changes (elimination of multi-relocatemode and replacement with change-group).
It also removes an extra re-export of EvacMode from Loader.hs....
htools: split tryMGAlloc in two
Currently, this function does both identify the best group for aninstance, and format the results (partially). The former functionalityis useful in other contexts too, so let's split the function in two.
htools: add node-evacuation of DRBD all nodes
This is much more complicated than either change primary (which istrivial) and change secondary (which is also simple).
The main strategy is:
- choose a new secondary (that will become the final primary) and...
htools: small optimisation in Cluster.collapseFailures
This makes fewer temporary data structures, and thus brings a bit ofimprovement in total runtime.
htools: add optional limit to allocation count
Currently, the tieredAlloc/iterateAlloc functions will not returnuntil the allocation fails; this means unit-testing theirfunctionality (e.g. that an instance can be allocated) is slow, sincethey will allocate all possible instances....
htools: add node-evacuation of DRBD secondaries
htools: rework compCV so that we can use [Node] too
In some cases, we have a [Node.Node] list, instead of a Node.Liststructure. Instead of converting [Node.Node] to the latter, let'srework compCV and add a compCVNodes such that we skip this(potentially expensive) step....
Adding a wrapper around "xm console"
The wrapper will connect to the console, and check in the background ifthe instance is paused, unpausing it as necessary.
Signed-off-by: Stephen Shirley <diamond@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Adding a wrapper around connecting to kvm console
Add opcode attribute for chained jobs
Design for chained jobs
htools: fix a doc bug in the hbal man page
Similar to the previous hspace fix.
Mark htools/README as obsolete
And point to the main INSTALL file for up-to-date information.
Fix htool unittest failure due to invalid prereqs
Currently, the test prop_Node_rMem does some reserved memory checks onnodes, and for the test to work correctly (can add more instances) werequire that the node free memory is greater than zero via aprerequisite....
htools: add parseUnit function
This is similar, but not identical, to Utils.ParseUnit. The biggestdifference is that we don't round up/down; as we only use integraltypes, the result will always be rounded down.
Moreover, since (real-world) disk sizes come in SI units, the function...
hspace: add support for units in the options
This patch converts hspace to allow units in all disk and memoryspecifications, and updates the man page accordingly. It also correcta mistake in the man page for the --max-cpu option.
hspace: also display the starting tiered specs
These are useful for checking what instance specifications we startwith (as they are not necessarily the first entry in the TSPEC map).
htools: abstract and simplify a multigroup function
In the future, this 'split/associated instance idx list' functionalitywill be used by other code, so let's move it to its own code.
The new method of computing the split also requires exporting one more...
htools: rename an internal constant
'emptySolution' is fine while we have only one Solution type(AllocSolution), but in the future we'll introduce another one andthus we should rename it.
htools: add a function for computing evacuated nodes
The new IAllocator interface is based on instances, not nodes, so weneed to backtrack and compute on which nodes we can't allocateinstances during the current operation. This patch adds a function for...
htools: add two monad-conversion utility functions
For monadic operations, these function will make it easier to convertother monads (or monad-like structure) to our 'Result' type.
htools: add more IAlloc data types and functions
These move the 'not supported' message deeper into the workflow;assuming that the workflow as added by this patch is correct, all thatremains is:
- for disk template DRBD8, plug in the currently existing mechanisms...
htools: add node-evacuate of DRBD8 primaries
htools: simplify select tests
Rather than running all possible tests in one go, split them out inthree cases: "normal", undefined default value, undefined list values(both of which require a valid response before them).
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Set startup_paused to False when restarting
This fixes the lint error:
E1120:1220:InstanceReboot: No value passed for parameter'startup_paused' in function call
htools: lookup instance names in select/exclude
Currently we just match the passed name with fqdns. With this change wedo a proper search for either the fqdn or the initial name component, asin Ganeti itself.
htools: Add if' and select as functions
if' is just a functional version of the if-then-else construct. Using itwe can manipulate ifs as functions, and also build other functionality.
select allows to choose a result depending on truth value of statements,...
htools: infrastructure for looking up names
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: remove obsolete option INodes
hspace no longer uses it, hence it can be removed.
hspace: also report the disk template
hspace: ask for the disk_template not number of nodes
This makes more sense, as the disk template is more meaningful thanthe number of nodes.
htools: add an utility function for disk templates
In the future we'll use this instead of requiring the number of nodesfrom the user.
htools: define the --disk-template option
Also sorts the export list correctly.