N+1: verify using minimum memory
Since instances can be started, failed over and migrated with less thantheir maximum memory N+1 will use the minimum memory for verification.
Note that this accounts only for the instances being moved beingresized, and not for the ones already on the node, as ganeti will not...
admin.rst: add information about maxmem/minmem
- Specify how maxmem and minmem are used at startup, failover and migrate- Add a paragraph about how to manually change the runtime memory, and what Ganeti will and won't do automatically as of version 2.6...
walkthrough.rst: updates for maxmem/minmem
- Fix two example outputs- Improve N+1 resolution section specifying how different memories affect N+1 calculation depending on the node
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix integer overflow in Node unittest
When using a really random "extra cpus" variable, this can overflow(especially on 32 bit) and thus the final value can be < 0, thus thetest will fail to fail.
It doesn't make sense to test with huge values, so let's just generate...
Use allow_runtime_chgs in cmdlib
- Report the attribute value in the hooks runtime variable.- Use it to abort migration if not enough memory is available on the target node.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
cli/opcodes: add a --no-runtime-changes migrate opt
This will be used for now to avoid ballooning memory at live migration time.
Migrate/Failover: allow less-than-max-mem op
Failover works magically after we relax the check: the start operationwill just use less memory, if less memory is available.
For migrate we need to balloon down the source memory before themigration, if the target node doesn't have enough capacity....
Failover and Migrate: acquire node resource locks
cmdlib._CheckNodeFreeMemory: return free memory
doc/rapi.rst: Document ipolicy parameter
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add vcpu_ratio definition to the IPolicy type
This will be "naturally" read in the JSON-based backends (Luxi/Rapi),and we add explicit code for handling it in the Text backend.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Remove Node.mCpu, replaced with the ipol vcpu_ratio
Since the iPolicy contains the vcpu_ratio now, we can remove theper-node mCpu value (which was the initial way to set this attribute).
Furthermore, we need to re-set the hiCpu value when changing thepolicy....
Temporary fix for hspace's NPU calculation
hspace computes the NPU (normalised CPU units) via the(used|pool|avail) VCPU divided by the (currently cluster-wide) maxvcpu_ratio.
However, in the future we'll have variable vcpu_ratios, so we can'tthe calculation as such. We temporarily change to computing the...
Change meaning of the --max-cpu option
Instead of always overriding, we should update the nodes's maxcpusetting only when the option is specified. Otherwise this will be readfrom the cluster or from the Ganeti constant for non-live backends.
Signed-off-by: Iustin Pop <iustin@google.com>...
Remove obsolete constant for vcpu_ratio
This is superseded now.
Re-indent code for readability
Otherwise the docstrings for members take too many lines.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Track 'normalised' used CPUs in Cluster.CStats
This will be used in hspace for proper tracking of used/pool/unavailnpus.
Add a new type for holding allocation statistics
The current RSpec type doesn't cut it anymore (and it's used in otherplaces), so let's introduce a separate type for holding thesestatistics.
Switch over some hspace stats from RSpec to AllocInfo
This patch changes the allocation delta stats from RSpec toAllocInfo. There's nothing interesting about it, but it should allowmore correct computations of Npus in hspace.
I also moved the AllocStats type alias from Cluster.hs to Types.hs...
Expand the tiered alloc unittest to check allocation stats
This was missing before, so with the new metric (NCpus), it's a goodtime to test this.
Fix gnt-cluster modify check of passed options
… and add the disk_templates/vcpu_ratio new entries. Since the defaultispecs_* are dicts, we can't compare them to none.
Additionally, add a QA test for this case (empty cluster modify).
Export the cluster ipolicy in iallocator
And remove a strange comment (the commit which introduced it,d61df03e, doesn't explain it). Also document these new keys iniallocator.rst.
Add ipolicy_vcpu_ratio to command line
This allows changing this value from the command line.
Add new ipolicy parameter vcpu_ratio
This adds the new parameter and changes the validation routines tohandle IPOLICY_PARAMETER value as floats. Very very ugly code :(,should be redone much cleaner.
Clarification on what objects.InstancePolicy is
Rename IPOLICY_PARAMETERS to IPOLICY_ISPECS
As opposed to the initial situation, the PARAMETERS will be more thanjust the min/std/max ispecs. Let's rename it, before re-adding thePARAMETERS constant with a different meaning.
Remove two extraneous imports
gnt-instance recreate-disks: Allow specifying new size
With this change a new disk size can be specified when recreating disks.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
LUClusterSetParams: When ipolicy is updated warn for new violations
We need to loop over all node groups to apply their modification to makesure that the new ipolicy actually affects their instance
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>...
LUGroupSetParams: Warn if the new ipolicy will violate some ipolicies
If we modify the ipolicy we need to verify if any instance would nowviolate the new ipolicy and warn about it. Any instance which alreadyviolates the old one is ignored.
Signed-off-by: René Nussbaumer <rn@google.com>...
cmdlib: Add a helper function for a set of violating instances
This helper will be used with the Op*SetParams and ipolicy verifications
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Rename ispecs-disk-templates to ipolicy-disk-templates
The ispecs (min/std/max) are different from the other ipolicyvariables (disk templates for now, more in the future). As such, let'srename the ispecs-disk-templates to ipolicy-disk-templates forconsistency with this logical difference....
fmtjson: A tool to format JSON data
While debugging an issue recently I had the pleasure of looking at along, single-line string of JSON data. This simple utility makes itpossible to format such data into a readable format.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Implement rbd disk template
Introduce the rbd disk template, which handles provisioning andmanagement of instance disks as block devices mapped to rbd volumeson a RADOS cluster.
The rbd template is of type DTS_EXT_MIRROR, meaning rbd-basedinstances can failover or migrate to any node, without...
Open VDI Port when spice_use_vdagent is enabled
To use the features spice agent provides, a dedicated virtio-serialchannel must be created in qemu-kvm. The communication between the agentand the other spice components takes place over this channel.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Fix iallocator group objects
… this is yet another manual serialisation of objects :(
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix upgrading the cluster ipolicy
When running from git HEAD, just initialising the ipolicy whennon-existing is not good enough; we need to upgrade also missing keys,etc.
This has downsides though, as it means we'll always 'rewrite' theipolicy, which might create silent data loss for unhandled changes...
LUInstanceRecreateDisks: Actually acquire node resource locks
Before this change an assertion would fail as the resource locks weren'tactually acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
cmdlib._CheckInstanceState: Use instance name in error message
The variable “instance” is an instance object, so to not get aserialized version of the instance, the “name” attribute must beused.
Unbreak build after merge 2ea64524ce
gnt-cluster command: Add option to prepend node name to output
With this patch a new option (“-M”, like dsh) is added to “gnt-clustercommand” to prepend the node name to all output lines.
$ gnt-cluster command -M uname -a | grep Linuxnode18.example.com: Linux node18.example.com [...]...
cmdlib: Drop use of “len(…) != 0”
Found by Constantinos Venetsanopoulos.
Merge branch 'devel-2.5'
Conflicts: lib/cli.py - trivial lib/cmdlib.py - merge new condition
Merge branch 'stable-2.5' into devel-2.5
cli: Log fully quoted command line
Instead of logging the concatenated arguments, with this patch everycommand will properly quote its full command line. This makes it easierto just copy-and-paste commands from “commands.log” to re-run them. Alsoremove a case of “x = a or b” for non-boolean values....
Migrate: don't check for free memory on cleanup
Cleanup just updates the config with the correct location of theinstance, or informs of its down status, but never starts it. As suchthere's no point in checking for enough free memory. Actually this check...
LUInstanceStartup: check for minimum memory
hypervisors: start instances with max available mem
If we have available more than MINMEM but less than MAXMEM, start a newinstance with that value, rather than MAXMEM.
Instance startup: lock primary node
This will be used to balloon other instances on the node if needed.
kvm: always start in paused state
Currently kvm starts in a frozen state if cpu pinning is set.This is useful for other purposes as well (eg. ballooning before theinstance uses the memory), so we move the functionality out of the cpupinning code.
Note that before the "continue" command was executed in a finally after...
Revert "cli: Disable abbreviation matching for options"
This reverts commit 232aab3f4f602a19f1226e85c3a3ecb245d60af4. Weshouldn't change the parsing of command line options in the middle ofthe 2.5.x series.
Fix wrong option names in QA and cluster-merge
Fix a bug in command line option parsing code
Fix bug affecting command line options of "keyval" type. Althoughescaping commands with \ is supported, it is is not applied to theinput recursively.
Signed-off-by: Nikos Skalkotos <skalkoto@grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>...
Add UnescapeAndSplit unittest for multi-escapes
This would have caught the bug in the first place. Argh,hand-generated test cases!
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Bump version to 2.5.0~rc5, update NEWS
cli: Disable abbreviation matching for options
Python's “optparse” module does option name prefix matching by default.Since this can lead to confusing behaviour, e.g. by specifying “--force”for a command which only has a “--force-multi” option, this patch...
qa: perform burnin with max/min mem from config
Burnin: test set-mem
Add new --maxmem-size and --minmem-size options.The old --mem-size option allows setting both at the same time.If --minmem-size is lower than --maxmem-size execute a runtime memorychange on all instances, bringing them to their minimum memory size....
gnt-instance modify -m|--runtime-memory
Implement memory ballooning in xen
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Implement memory ballooning in kvm
xen: add the "maxmem" parameter in instances config
This is set to the same amount as the memory for now, but will allowstarting instances with less memory than their maximum.
Implement memory ballooning in the fake hv
No-op memory ballooning in chroot and lxc
Currently they have no memory limits set.
Add instance_balloon_memory rpc
Hypervisor: allow ballooning of instance memory
This new hypervisor call sets the memory of an instance to a new value,through ballooning.
Accept both PUT and POST in noded
Currently, noded requires PUT, even though the semantics of the RPCcalls do not match a PUT. We change the code accept both PUT and POST,with the intention to remove the PUT support in a later version.
On the client side, the RPC code is changed to send POST requests....
Rework instance rename QA tests
After looking again at the QA speeds, it turned out that instancerename was run 12 times via the command line and 12 times via RAPI;outside of start/stop which we need for maintenance during QA, this isthe most used test....
Another small optimisation in test data generation
It seems that QuickCheck will happily generate huge data structures,if that's what one requests. For this test, where we ask for [[Char]],it could happen that we get a huge number of chars, so let's limit...
Generate tags using the helper in Node tests
Otherwise we test very big lists (slow) without any advantage in doingso.
Combine two Node unittests
These two tests are very similar, so let's run them together insteadof generating the data twice. Also improve the test a bit.
Add unit-tests for --help/--version as common options
After reorganizing the way we process options, we can finally add atest that all program personalities accept --help and --version andthat they exit early for these two.
This trivial patch has a higher than expected impact on the coverage,...
Split the list of program personalities into a module
… a mini-module, actually. This is only done for unit-testing, as wedon't want to duplicate the personalities list in the tests.
Parse options in htools.hs, not in each program
This patch moves the call to getArgs/parseOpts into htools.hs, asopposed to the current situation where htools.hs only calls the mainfile of each program, which in turn calls these two functions.
By doing it only once we remove some duplicate code and we also export...
Add some very basic options test
Just tests that passing a string argument returns it correctly. We'renot interested in the flags themselves that much just that thefunction itself parses some options at least correctly.
Split CLI.parseOpts into IO/non-IO parts
This will allow unittesting the CLI options much more easily. Thepatch also fixes an inconsistency: usage help was displayed with anextra new line in the error case (but not in --help).
Add test for CLI.parseYesNo
Also fix a tiny typo in the docstring.
Fix bug in CLI.parseISpecString and add test
'zip' finishes on the shortest list, so parseISpecString accepted anystring of 3 values or more, which is not good. Fix this and add testfor both successful and failed parsing.
Add a unittest for the 'Simu' backend
This only test successful parsing, but for this case it testsreasonably well the resulting cluster properties, I think (coverageis >70% for expressions).
Further optimise instance test data generation
Following on the lessons from this week (commit 7806125e), this patchreplaces generation of instances `suchThat` they are smaller than onnode with direct generation of instances with upper bounds onmemory/disk/cpu....
Add a Makefile target for profiling targets
This simplifies the building of profiling-enabled htools binaries;there are two targets, one used for the initial two-stage build, andone that can be used for incremental builds.
The patch adds a new internal variable for the htools/test binary,...
gnt-instance: Adapt replace-disks to the new opcode
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
TLReplaceDisk: Add ipolicy checks
Fix very slow unit-test data generation in some cases
Learning, learning. It turns out that for objects with even just fivefields, generating them and filtering can be costly, especially whenthe filtering is nested (as it was in this case, arbitrary1 →...
Fix one bad unit-test data generation case
It could happen that the node is generated with 0 disk available, inwhich case the operation will fail due to disk issues, not CPU. Fixthis by using the recently-introduced genOnlineNode helper.
Further JSON unit-tests for various Types.hs types
Add a unit-test helper function
… instead of always using "printTestCase … False".
Extend node evacuation unittests
Currently they only check for change secondary, via iallocator'srelocate method. We add two separate unittests that checks all thenode evacuate modes (ChangeAll, ChangePrimary) and also the groupchange request. All of these are done only for DRBD instances....
Fix a few issues in unittest data generation
This patch fixes two ugly but non-critical issues:
- the getFQDN wrongly generated only the first component as limited-length name, the rest could be any length, and that makes the generated name bad for display purposes...
Remove some dead code
While wondering why Cluster.tryReloc is not unit-tested, I realisedthat this is no longer uses. Apparently, last summer's new IAllocatormodes and related changes have silently deprecated this function.
Fix another data generation issue in tests
Another "good" seed value manages to generate nodes with identicalnames, which of course makes the sets be of smaller value than thelists. Avoid this issue by generating unique node names.
Further fixes to instance policy validation
As a followup from "Remove extraneous check in policy creation", thereare more places where we build an ipolicy, and then manually check forits validity. This is very bad style, as it duplicates theverification code across many places....
Add a help function for generating test nodes
This abstracts the concept of "give an sane, online node" in a singleplace, rather than spread around the tests. We also change thesignature for isNodeBig for easier usage.
Generate clusters with unique node names in tests
Nothing breaks otherwise in the cluster routines, but serialisationto/from text gets confused otherwise.
Add a complex allocation/serialisation/load test
This is unfortunately a complex test that checks if:
- starting from a empty cluster- allocating a few instances on it- serialising it via the Text backend- loading it back into internal data structures...
Implement saving/restoring of policies in Text backend
This adds a new optional (so we're backwards-compat) section to thecluster data so that we can reliably save/restore the policy data.
Note that different from the other sections, we save all policies...
Further unittests for ipolicy/ispec text serialisation
Fix a couple of bugs in handling offline instances
The addition of the offline state for instances has introduced a fewbugs related to their handling. The current code adds a helperfunction for making the check whether an instances uses reservedmemory on the secondary (and uses it instead of just autoBalance),...