Add missing Constructor for SetParamsMods
Disks and nics can not only be addressed by indices, but alsoby name. Hence add a constructor for this case as well, to befaithful to the python world.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Ignore hlint warning "Error: Too strict if" in Server.hs
A previous patch [229da00] added an annotation for ignoring the warning,but to the middle of a function, which doesn't compile. This patch movesthe annotation to the end of the function to correct the problem....
This warning appears only in newer versions of hlint (mine was v1.8.43),and in this case it's reported incorrectly. The arguments to "showJSON" have different types, therefore it's not possible to move "showJSON" in...
Document incompatible change of luxi
With the change to luxid handling job queue requests, wealso inherited the parsing semantics of luxid, i.e., arequest is only considered syntactically correct, if allparts are. Document that we decided to consider this change...
Adapt TestJobSubmission to new semantics of SubmitManyJobs
Historically, Ganeti would partially handle syntactically incorrectluxi requests, if the outer structure was that of SubmitManyJobsrequest. Moreover, this behavior was verified by a test. We changed to...
Update future upgrade plans
Now that mandatory opcode fields are mandatory also on disk,document that, once we use soft drains during upgrades, the diskrepresentation of the job queue needs to be updated by cfgupgradeas well.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Add a constructor to BaseOpCode
In order to guarantee all mandatory slots to be present, adda custom constructor to BaseOpCode adding those fields withdefault value, instead of inheriting the constructor fromoutils.ValidatedSlots.
FillDict: do not use exceptions for flow control
While it's generally better to beg for forgiveness than ask forpermission in Python, it's not a good idea to rely on exceptions tomodel control flow, especially if the resulting code is simpler.
Signed-off-by: Santi Raffa <rsanti@google.com>...
Remove hardcoded references to File, SharedFile templates
DTS_FILEBASED is a constant that exists and this commit makes surethat it is used whenever sensible, rather than resorting to hardcodingthe pair of templates in very many files.
Make masterd call luxid to write jobs to queue
As the responsibility for writing the job queue changedto luxid, make masterd call to luxid when it is necessaryto write a job.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Set reason trail for derived opcodes
As the base parameters also includes the reason trail,set it, if not set already, together with the otherbase parameters in the case of an opcode expanding toa list of opcodes.
Make luxid handle SubmitManyJobs
Handle this request by writing the jobs to thequeue and inform masterd; masterd will then alsodistribute the jobs to all master candidates.
Make luxid handle SubmitJob
As luxid is to take over responsibility for the job queue,handle this request by writing the job to the queue and theninforming masterd; masterd will also distribute the job toall master candidates.
Add the predicate of the queue being open
Adding jobs to the queue is only allowed if the queueis not drained.
Provide path to the queue drain file
Since luxid is going to write to the job queue,it needs to honor drains of the queue as well.
Add Luxi Request to pick up a job in the queue
During the transition to the new daemon layout, from step 2onwards, luxid will write to the queue but masterd will triggerthe execution. Therefore, add a new luxi request to tell masterdto pick up a job that has already been written to the queue....
Factor out functionality to pickup a job from the queue
The current restart procedure for masterd includes functionalityto pick up a job from the queue and restart it, if it hasn't beenstarted before. Move this functionality into a separate functionto be able to have the enqueuing be done by luxid....
Provide means of locking a file
Two avoid two processes simultaneously accessing the sameon-file structure, like the job queue, file locks are used.Therefore, provide this functionality in Haskell as well.
Provide path to the queue lock file
To avoid several processes accessing the queue at the same time,Ganeti locks the queue via a lock file on disk. Provide the pathto this file.
Provide a function to write jobs to disk
This function writes a (non-archived) job to disk. The filename can be computed from the job id, which is part of the job.
Provide means to allocate new job ids
Add utility functions to allocate new job ids by increasingthe value stored in the serial file. As this function isused in a multi-threaded program, synchronize access overan internal lock.
Add function to get master candidates from configuration
With luxi daemon taking over part of the job queue management, it willalso be responsible for replicating the queue to all master candidates.Therefore, add a function to extract the list of master candidates from...
Support RPC asking to replicate part of the job queue
To be able to replicate the job queue, in particular the serial,luxid needs to be able to send the jobqueue_update RPC. So addits definition.
Provide function to obtain the unique element of a list
This version of 'the' properly lives in the 'Result' monad,as opposed to traditional one calling 'error'. The reasonwhy it is 'Bad' that not precisely one element is returnedis given as argument....
Provide method to read job serial number
This methods allows reading the maximal job serial number fromdisk.
Provide convenience function to create Job from op-codes
This function handles the pure part of generating a job,i.e., assuming the job id already assigned and not settingtime stamps.
Add function to resolve dependencies in meta op code
When queueing many jobs, the dependencies between them need tobe resolved with the knowledge of their respective job id.Lift the computation of the absolute dependency to the levelof MetaOpCodes.
Add function to compute the absolute id of a dependency
SubmitManyJobs also accepts jobs with dependencies given asrelative ids. Together with the absolute id of the job, onceassigned, the dependency can be resolved. Add a function doingthis computation....
Provide a convenience method to optain a QueuedOpCode
When generating jobs from sequences of op-codes, it is necessaryto wrap op-codes into queued form.
Add utility function tryAndLogIOError
This function allows to use 'IO a' objects in a safeway, using the 'try' function; the outcome is reportedas a 'Result'. IOErrors are logged and the result is'Bad', while in the case of no errors, a result-yielding...
Provide utility to atomically write a file
To keep our on-file data consistent at any moment, we changefile contents by atomically replacing the file with a new one.
Design document for custom SSH ports
See issue 235. It turned out the feature is a bit more complex than expected,hence this design doc.
Signed-off-by: Petr Pudlak <pudlak@google.com>Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Optimize Haskell compilation
The idea is to compile (on demand, that is, when necessary) eachsource file to a normal object file, a coverage object file, aprofiling object file, and a test object file. Also, a given Haskellbinary is linked with the proper object files. This is achieved with...
Fix getNodeRole
In the configuration, the master node is now givenby its uuid. Therefore, compare the uuid and not thename to find out if a given node is the master.
Merge branch 'stable-2.10' into master
Merge branch 'stable-2.9' into stable-2.10
Conflicts: NEWS: trivial configure.ac: drop suffix bump of stable-2.9
Revision bump for 2.9.0
Schedule release of 2.9.0
...and mention the last change pulled in from stable-2.8.
Merge branch 'stable-2.8' into stable-2.9
Conflicts: lib/cmdlib/instance_storage.pyResolved by manually applying the node name to uuidtransition on the version of stable-2.9.
Improve error message for replace-disks
In some conditions, replace-disks will fail if the disks are not properlyactivated. Improve the error message suggesting to run activate-disks beforeexecuting replace-disks.
Fixes Issue 606.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Fix documentation
Fix documentation in constants containing values in seconds.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Replace all constant definitions with re-exports
Remove all constant definitions in 'lib/constants.py' and re-exportthe module 'lib/_constants.py', which contains the generatedconstants.
Prepare constants for automatic reexport
Hs2Py constants: 'hvsParameterTypes' and 'hvsParameters'
Add constants 'hvsParameters' and 'hvsParameterTypes' to the Haskellto Python constant generation.
Add visual/color marks to the QA script
Add the function name to textual descriptions of tests
It makes it a bit easier to find the corresponding functions.
Fix indentation that triggers PEP8 error
hsqueeze: document expansion planning in man page
Describe the new behavior of hsqueeze to also plan taking nodes onlinein the man page.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add a test for hsqueeze expansion
In this example, there are 7 nodes, each with capacity for4 instances. Two nodes are online and host 4 instances each,thus forming a cluster with no free resources. Four additionalnodes a standby, and two additional node is offline for other...
hsqueeze: support planing for onlining nodes
If the amount of free resources falls below a given threshold,hsqueeze will suggest putting standby nodes back online untilthe minimum of free resources is reached, or all standby nodesare online.
Add an --minimal-resources option
Add a new option, to be used by hsqueeze, to specifythe amount of free resources that has to be on eachnode, in order not to start onlining standby nodes.It is given as a multiple of the standard allocation,as specified by the instance policy....
Text Backend: correctly read data for offline nodes
With standby nodes, simply ignoring the specification ofan offline node is not sufficient any more.
hsqueeze: add a test for the -v option
This test also verifies that the master node isnot considered for being offlined.
Change default for target resource to 2.0
The target resources, as used by hsqueeze, are supposed tobe strictly higher than the minimal resources. However,keeping minimal resources of less than a single instanceis not a useful reserve.
git-ignore test/hs/hsqueeze
configure: avoid duplicate Haskell module checking
There are some Haskell modules, that are optional for production,but needed in test. Check for those only once, by unconditionallychecking before considering the modules for the specific feature.
Check if hotplug is supported in CheckPrereq
Introduce new RPC hotplug_supported that invokes the correspondinghypervisor's method which checks if hotplug is generallysupported. Call this RPC early in CheckPrereq() and abort ifhotplug is not supported. Currently only KVM hypervisor with...
Conflicts: configure.ac: take both additions (and fix)...
Remove FIXME + update man pages wrt iallocator
This patch removes a FIXME regarding the retrievalof the disk template for computation of the clusterinformation for the iallocator. After examining thecode, I figured out that it is actually a quiterobust way to just check if the disk template is...
Add all dependencies for confd as test dependencies
Since our tests pull in confd as a dependency, all build dependenciesfor confd are also necessary to run the tests.
Don't check for master netdev when using OVS
This patch disables checking if the given master_netdev exists during clusterinit.When OVS is used, the needed Open vSwitch will be created later on (withOpClusterPostInit) and is thus not available at this point of time....
Fix TypeError in backend/ConfigureOVS
In case of running into a failure during Open vSwitch creation, the error isnot given, but a TypeError is thrown. This is because the number of argumentsfor the format string was not equal to the place holders.
This fixes the issue....
Create Open vSwitch on Master during Cluster init
Currently, Open vSwitch is only created on additional nodes, not on the masteritself.This fixes the issue and creates and configures an Open vSwitch on the masternode during cluster init.
Signed-off-by: Sebastian Gebhard <sege@fs.ei.tum.de>...
Add design document for multi-version tests
This patch adds a design document for multi-version tests, discussingwhat they aim to do and how they should be implemented.
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
SimpleRetry on BlockDev.Remove()
Sometimes, upon disk removal, corresponding file descriptorsare kept briefly open by various processes (hypervisor, blkid, etc.).With this patch, we retry several times before raising the appropriateerror, thus making disk removal more robust against those corner cases....
Add snap-server to the test-relevenat packages
While snap-server is only needed for the optional monitoring daemon,some tests, notably those testing these optional features, still dependon it. So, if snap-server is missing, the Haskell tests should not be...
Mention compression for moves/backups in NEWS
Mention the possibility to specify compression for transfers / instancestorage in NEWS file.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add --compress option to instance_move
instance_move now supports the --compress option with which it'spossible to select the compression mode to use during the instance move.
Add possibility to compress to OpInstanceCreate
OpInstanceCreate now supports the 'compress' option. It allows to enablecompression during instance imports.
Honor compress parameter in remote backups
Use the specified compression mode not only for local exports, but alsofor remote ones.
Add --compress option to gnt-backup import
Add a new command line parameter to gnt-backup import and passits value to OpInstanceCreate. Also document the option in theman page.
Add --compress option to gnt-backup export
Add a new command line parameter to gnt-backup export and pass its valueto OpBackupExport. Also document the option in the man page.
Support import with compressed instance moves
Support compressing instance data while sending it to the target node oninstance imports.
Add local compression to OpBackupExport
OpBackupExport is extended by a compress parameter. This parameter(either 'none' or 'gzip') controls if instance disks are compressedbefore being sent over the network to the destination node.
Signed-off-by: Thomas Thrainer <thomasth@google.com>...
Add --compress option to gnt-instance move
Add a new command line parameter to gnt-instance and pass its value toOpInstanceMove. Also document the option in the man page.
Add possibility to compress to OpInstanceMove
OpInstanceMove now supports the 'compress' option. It allows to enablecompression for intra-cluster instance moves.
Remove trailing whitespace
Remove trailing whitespace from OpCodes.hs and OpParams.hs.
Remove blockdev_export RPC
This RPC call is no longer used, as for intra-cluster instance moves theimport/export daemon is used too.
Optimize dd parameters for instances moves
Benchmarks have shown that increasing the buffer size and disablingsynchronous output increased the throughput by about 10MiB/s, so makethose parameters the default.
Note that not using sync for output doesn't increase the risk of data...
Use import-export daemon for intra-cluster moves
This unifies the inter- and intra-cluster moves and the backup code, soless code needs to be maintained.
Also fix failing tests for LUInstanceMove by following the code changesin the tests.
Use cluster variable consistently
The cluster object is already retrieved from the configuration, so useit consistently in LUInstanceMove.CheckPrereq().
Correct comments
Correct comments which confuse the target node with the secondary nodeof an instance in LUInstanceMove.
Fix expectation of the htools wrapping
With the addition of hsqueeze, htools obtained a new personality.Additionally, the longer htool name also changes the line wrapping ofthe descriptions. Adapt the expectation accordingly.
Add a man page for hsqueeze
Describe the current state of the Ganeti dynamic power managementtool, hsqueeze, in a man page.
Add test for hsqueeze compression
In this example cluster, there are 6 nodes withcapacity for 4 instances each, holding a totalof 8 instances. So even after turning off 2 nodesand rebalancing, every node still has capacity for2 instances.
Add hsqueeze planning for compression
Add a new htool, hsqueeze, for dynamic power management. Thiscommit only implements the first useful part: plan (but notexecuting) taking nodes offline nodes still keeping within theresource limit.
Provide --target-resources option
Add a new option, to be used by hsqueeze, to specify thetarget free resources on each node. It is given as a multipleof the standard allocation, as specified in the instance policy.
Show the key in "'None' is not a valid Maybe value"
Currently the error message doesn't say which key is affected, whichmakes it kind of useless.
Add RpcResult.__repr__ for debugging
This way it's possible to easily log the whole result in a humanreadable form.
Let git ignore `.hsenv` directory and vim's session file
If a configuration file can't be loaded, show which one
So far the error message only says it can't be loaded, but not what fileit is.
Refactor `getAllConsoleParams` to be more readable
Instead of creating a lot of lists and then zipping them, just workon every item one at a time. It makes the code much more readable.This changes the ordering of "effects", but the "effects" here are just...
Fix usage of xm/xl for gnt-instance modify --new-primary
This fixes issue 519. When calling 'gnt-instance modify--new-primary', an exception was raised, because aRPC call was not called with the right hypervisorparameters which caused the backend function to crash,...
Hs2Py constants: add 'hvcDefaults'
Add 'hvcDefaults' to the Haskell to Python constant generation.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Hs2Py constants: add 'diskDtDefaults'
Signed-off-by: Jose A. Lopes <jabolopes@google.com>...
Hs2Py constants: add 'value*' and 'niccDefaults'