Add drbd_helper rpc call
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Check drbd usermode helper in cluster verify
Set drbd usermode helper on config upgrade
Generalize a recursive check on logical disks
Add drbd_usermode_helper to configuration
VerifyNode: add usermode helper reply
BaseDRBD: provide a way to query usermode_helper parameter
OpCreateInstance: do not require hv/be/os params
It is perfectly legal to create an instance using only defaults(although beparams will be most times passed in), so let's relax therequirement for these three parameters.
Signed-off-by: Iustin Pop <iustin@google.com>...
A few more type definitions
This is to simplify the type declarations in the actual LUs.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Make _CheckDiskTemplate a valid type checker
Rework the "type" system
This patch merges the _OP_REQP and _OP_DEFS class attributes into a_OP_PARAMS list, which holds both. The associated unittest checks thatall opcode attributes are declared and checked, and that no LU uses theold fields (could be removed later)....
Remove _CheckBooleanOpField
This is no longer used, and we can remove it.
Merge branch 'stable-2.1'
RapiClient: fix multi-authentication in Python 2.6
In Python 2.6 the urllib2.HTTPBasicAuthHandler has a "retried" count forfailed authentications. The handler fails after 5 of them. To solve thiswe reset the handler's "retried" member variable to 0 after every...
utils.OwnIpAddress: Change try/except for Python 2.4
Python 2.4 doesn't support “except” and “finally” in the same block.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
RAPI client: Switch to pycURL
Currently the RAPI client uses the urllib2 and httplib modules fromPython's standard library. They're used with pyOpenSSL in a very fragileway, and there are known issues when receiving large responses from a RAPIserver....
baserlib: Use boolean type for boolean variables
This does not yet fix all issues in the RAPI interface which wereintroduced with the type system. More testing is needed.
Fix check in gnt-instance modify -t
Currently when changing the instance disks the instance status (whichmust be down) is only checked if the target status is among the mirroredones. This is incorrect: the instance must be down either way, even ifthe desired final status is non-mirrored....
Adding check_name option to the opcode and luxi call for instance rename
This will allow instance rename without dns check as it does for instanceadd.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Enable utils.OwnIpAddress to work with IPv6 addresses
Currently a TcpPing using the target address as source is used todetermine wheter or not an IP address belongs to a node. This does notwork for IPv6 anymore (I'm not sure why) as binding to IPv6 localhost as...
Barebones LXC hypervisor
This needs lots of work, but it can successfully launch an LXC-basedinstance. See the docstring for the limitations/work to be done.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>
DRBD IPv6 support
Support IPv6 configuration for 'drbdsetup show' parser and add unittestsconcerning that case. Renames some data files to use consistent namesclarifying their usage.
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
IPv6 support for utils.TcpPing()
Add function to retrieve family of an ip address
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Introduce utils.IsValidIP{4,6}()
This patch introduces functions to check for valid IPv4 and IPv6addresses and converts IsValidIP() to return True if it is either a IPv4or a IPv6 address.
For now we do not change the functional behavior and replace IsValidIP...
Fix osparams bug when gnt-cluster modify'ing
Commit 625ac1135834c27a2e78fd61bf2df2ce29ad2966 introduced 'osparams'slot in OPSetClusterParams which was not passed when invoked viagnt-cluster modify.
Signed-off-by: Manuel Franceschini <livewire@google.com>...
Introduce IPv6 constants
Replace '0.0.0.0' with constant
Rename some constants to facilitate IPv6 support
Pass force variant option at instance creation
This was supposed to be done in"06073e857e3b518c7195d57306ae01793240c0c2" but by mistake the field wasadded to batch create rather than the normal instance create.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Fix a few uncommon pylint errors in compat.py
- It's ok if the optional modules functools, roman and hashlib are not there.- It's ok to reference any and all before defining them. We're doing all this exactly because, under python 2.4, they are undefined....
AsyncTerminatedMessageStream: send_message
This function adds the ability for a AsyncTerminatedMessageStream tohave a thread-safe message delivery function.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
AsyncTerminatedMessageStream: limit message count
Currently the message stream can process any number of messages inparallel (if they get dispatched to different threads or processes).In order to limit their number we only handle messages and read from...
Better specify old object slots removal policy
The comments seem to imply we can remove deprecated slots after 2.1 isreleased. This is not quite true, if we still want to support upgradingstraight from 2.0. So let's specify better that they can be removed only...
Fix a type declaration error in LUTestAllocator
Also, improve logging of type errors, to ease debugging.
Fix _UpgradeConfig docstring
jqueue: remove the _big_jqueue_lock module global
By using ssynchronized in the new way, we can remove the module-global_big_jqueue_lock and revert back to an internal _lock inside the jqueue.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
ssynchronized: act on a class member
The ssynchronized decorator takes the lock to act on in input.With this change we allow a string to be passed, and if so we assume thefunction it protects is a class method, and we act on the member of theclass itself named as the string we got....
Share the jqueue lock on job-local changes
We can share the jqueue lock when we do per-job updates. These onlyconflict with updates/checks on the same job from another thread (eg.CancelJob, ArchiveJob, which keep the lock unshared, since they are lessfrequent)....
_OpExecCallbacks abstract _AppendFeedback
Move some code to a decorated function rather than explicitelyacquiring/releasing the lock in AppendFeedback.
Remove utils.LockedMethod
All users of this function were converted to another method, hence wecan remove it.
jqueue: convert to a SharedLock()
Remove the jqueue _lock member and convert to a _big_jqueue_locksharedlock. This allows smooth transition from the old single lock to amore granular approach.
MarkUnfinishedOps: update job file on disk
Every time we call MarkUnfinishedOps we do it in a try/finally blockthat updates the job file. With this patch we move the try/finallyinside. CancelJobUnlocked is removed, because it just becomes a wrapperover MarkUnfinishedOps with two constant values....
Remove spurious empty line
Remove pred from compat.any/all
This makes it compatible with the python builtin, and we can even usethe builtin when running under the right version of python. The all andany functions are renamed to _all and _any, so that they can be tested,and (non)existing unittests are updated (translation: there are no unit...
Two more fixes for OS params and opcode defaults
If the OS is not using API v20, the parameter verification should beentirely skipped.
The second change is a simple typo.
Fix breakage due to OS parameters
I was using wrong python installation path (thanks Guido!), so I was notactually testing the new backend.py module. Two immediate things bugsare fixed, and after these burnin passes again…
Introduce constants.DTS_MAY_ADOPT
DTS_MAY_ADOPT include disk templates that may use disk adoption and will beused in all respective checks.
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Convert TestDelay repeat to the type system
Turns out with the current implementation of the type system a variablecan both be required and not required. Interesting (but perhaps worth afurther look).
Rename some constants and relax pylint rules
For the type system, we want a slightly relaxed rule for constantnaming, so we update the pylint rule. But the old _TPInt and _TNEStringwere not clear enough, so we expand them.
Introduce a micro type system for opcodes
Currently, we have one structual validation for opcode attributes: the_OP_REQP, which checks that a given attribute is not 'None', and therest of the checks are done at runtime. This means our type system hastwo types: None versus Not-None....
LU.CheckPrereq: do not require implementation
Currently, the base class LogicalUnit's CheckPrereq will raiseNotImplementedError, which means that the child LUs have to implementit. However, many LUs don't actually have a need for this function(hence the many "pass" statements as the only body)....
Some more CheckPrereq/CheckArguments cleanup
For a few LUs, a few tests in, or even the whole CheckPrereq, can bemoved to CheckArguments, as they don't touch state and only do a 'type'validation.
Add OS verification support to cluster verify
For this, we needed to extend the NodeImage class with a few extravariables, and we do a trick in the node verification where we pick thefirst node that returned valid OS data as the reference node, and then...
Move opcode attribute defaults to data structures
LUExportInstance had two opcode fields set to default via both_CheckBooleanOpField and getattr(…, False).
Cleanup LU.ExpandNames versus CheckArguments
When LogicalUnit.CheckArguments was introduced, not all code dealingwith static argument checking was moved to it; many of these checks wereleft in ExpandNames. With time, most of them migrated, and this patch...
Abstract export mode validity check
The export mode is checked in two places with the exact same code…
RAPI: switch evacuate node to the new model
This patch removes the last use of the old-style OpEvacuateNode. It alsofixes the dry-run mode for this RAPI resource - the dry-run parameterwas not used at all before.
Remove the obsolete EvacuateNode OpCode/LU
All code has been switched to the new-style LU… time for cleanup.
Add OS parameters to cluster and instance objects
The patch also modifies the instance RPC calls to fill the osparameterscorrectly with the cluster defaults, and exports the OS parameters inthe instance/OS environment.
Show OS parameters in cluster/instance info
Add support for OS parameters during instance add
This is not yet complete, as it lacks proper support for instanceimport.
_GetUpdatedParams: enhance value removal options
This patch adds controls for whether we recognizeconstants.VALUE_DEFAULT or not as a default value, and also addsdash-prefixes as another way for parameter removal.
Add support for modifying cluster OS parameters
We use _GetUpdatedParams in order to support removal too, and thenvalidate the OS parameters if the OS exists.
Add support for modifying instance OS parameters
We move the instance OS rename checks earlier, as we need to run thevalidation against the new OS, if it has changed.
Add support for OS parameters during import/export
Nothing special here, just copy/adjust the beparams code.
LUDiagnoseOS: add more fields, cleanup
This patch exports all the way from backend a new field ‘api_version’which holds the list of support API versions, and exposes the (alreadycomputed) ‘parameters’ field.
The patch also reworks (again) the field calculation in its Exec()...
Silence a pylint warning
The OS parameters code will bump the number of lines over 10K, and thuswe need to silence this (no, we don't want any other module to becomethis big…, so we use a targeted silence only).
Add os api v20 and related fields to the OS object
Add reading of OS parameters from disk
The patch also modifies the internal methods in LUDiagnoseOS and gnt-osto deal with the format change of call_os_diagnose.
Introduce an RPC call for OS parameters validation
While we only support the 'parameters' check today, the RPC call isgeneric enough that will be able to support other checks in the future.The backend function will both validate the parameters list (so as to...
Remove job object condition
We don't need it anymore, since nobody waits on it.
Parallelize WaitForJobChanges
As for QueryJobs we rely on file updates rather than conditionnotification to acquire job changes. In order to do that we use thepyinotify module to watch files. This might make the client a bit slower(pending planned improvements, such as subscription-based...
Update the job file on feedback
This is needed to convert waitforjobchanges to use inotify and theon-disk version and decouple it from the job queue lock. No replicationto remote nodes is done, to keep the operation fast.
Don't lock on QueryJobs, by using the disk version
We move from querying the in-memory version to loading all jobs from thedisk. Since the jobs are written/deleted on disk in an atomic manner, wedon't need to lock at all. Also, since we're just looking at the...
Add JobQueue.SafeLoadJobFromDisk
This will be used to read a job file without having to deal withexceptions from _LoadJobFromDisk.
jqueue._LoadJobFromDisk: remove safety archival
Currently _LoadJobFromDisk archives job files it finds corrupted. Sincewe want to use it to load files without holding locks, this could causea conflict: we just move the feature to _LoadJobUnlocked which is always...
Add repetition count to the TestDelay opcode
If the repetition count is not passed or is passed as 0 we sleep exactlyone time, otherwise we sleep "repeat" times and log in between.
Merge branch 'devel-2.1'
Add "adopt" to the allowed disk parameters
"adopt" was missing from bd061c3, thus breaking disk adoption.
Fix warnings with Python 2.6
'format' is a new built-in function, and 'bytes' is a new builtin type.We rename this to make pylint happy (and remove potential bugs).
Fix a small bug introduced in cf26a87a
Commit cf26a87a added a tiny typo, which would break non-FQDN argumentsto modify node storage.
Fix the type of 'valid' attribute in LUDiagnoseOS
The update of the valid status in LUDiagnoseOS says:
valid = valid and osl and osl[0][1]
However, in Python, “True and []” (which '[]' we get for an invalid OS)will result in “[]”, and thus the valid field for an OS will be either...
Fix handling of errors from socket.gethostbyname
Socket functions can raise more than just gaierror. Most of the times,socket.gethostbyname_ex will return gaierror, but rarely it will alsoraise herror. For completeness, we catch all socket exceptions with data...
Workerpool.AddManyTasks: check tasks type
Each task has to be a sequence, or the RunTask call will fail.
jqueue.AddManyJobs: use AddManyTasks
Rather than adding the jobs to the worker pool one at a time, we addthem all together, which is slightly faster, and ensures they don't getstarted while we loop.
RAPI client: Add support for Python 2.6
The httplib module used by urllib2 requires its sockets to have amakefile() method to provide a file-like interface (or ratherfile-in-Python-like) to the socket. PyOpenSSL doesn't implementmakefile() as the semantics require files to call dup(2) on the...
Bump RPC protocol version to 40
Many RPC calls have changed in Ganeti 2.2, hence bumping the RPC protocolversion.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix parameter names in SimpleFillBE/NIC docstrings
WorkerPool.AddManyTasks
Useful if we want to add many tasks at once, without contention with theprevious one we added starting.
AsyncAwaker: use shutdown on the socketpair
This makes sure the out_socket can only be used for writing, and thein_socket for reading.
jqueue: make replication on job update optional
Sometimes it's useful to write to the local filesystem, but immediatereplication to all master candidates is not needed.
The _WriteAndReplicateFileUnlocked function gets renamed to_UpdateJobQueueFile, as calling "write and replicate, but don't...
s/queue._GetJobInfoUnlocked/job.GetInfo/
The job queue currently has a static _GetJobInfoUnlocked method.Changing it to be a normal method of _QueuedJob, which makes more sense.
Abstract loading job file from disk
Move the work from _LoadJobUnlocked to _LoadJobFileFromDisk, which canthen be used in other contexts as well. Also, if we fail to deserializethe job, archive it as well (before we archived it only if we failed tocreate the related object, but kept it there if deserialization failed....
jqueue: simplify removal from _nodes
Somewhere we do try/del/except and somewhere just pop. Using popeverywhere saves lines of code.
ListVisibleFiles: do not sort output
Among all users, turns out just one may need the output to be sorted.All the others can cope without.
Remove a TODO
Since OS objects are not stored in the configuration, we cannot putos_hvp there, therefore the TODO is obsolete…