Fix _UpgradeConfig docstring
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
jqueue: remove the _big_jqueue_lock module global
By using ssynchronized in the new way, we can remove the module-global_big_jqueue_lock and revert back to an internal _lock inside the jqueue.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
ssynchronized: act on a class member
The ssynchronized decorator takes the lock to act on in input.With this change we allow a string to be passed, and if so we assume thefunction it protects is a class method, and we act on the member of theclass itself named as the string we got....
Share the jqueue lock on job-local changes
We can share the jqueue lock when we do per-job updates. These onlyconflict with updates/checks on the same job from another thread (eg.CancelJob, ArchiveJob, which keep the lock unshared, since they are lessfrequent)....
_OpExecCallbacks abstract _AppendFeedback
Move some code to a decorated function rather than explicitelyacquiring/releasing the lock in AppendFeedback.
Remove utils.LockedMethod
All users of this function were converted to another method, hence wecan remove it.
jqueue: convert to a SharedLock()
Remove the jqueue _lock member and convert to a _big_jqueue_locksharedlock. This allows smooth transition from the old single lock to amore granular approach.
MarkUnfinishedOps: update job file on disk
Every time we call MarkUnfinishedOps we do it in a try/finally blockthat updates the job file. With this patch we move the try/finallyinside. CancelJobUnlocked is removed, because it just becomes a wrapperover MarkUnfinishedOps with two constant values....
Remove spurious empty line
Remove pred from compat.any/all
This makes it compatible with the python builtin, and we can even usethe builtin when running under the right version of python. The all andany functions are renamed to _all and _any, so that they can be tested,and (non)existing unittests are updated (translation: there are no unit...
Two more fixes for OS params and opcode defaults
If the OS is not using API v20, the parameter verification should beentirely skipped.
The second change is a simple typo.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix breakage due to OS parameters
I was using wrong python installation path (thanks Guido!), so I was notactually testing the new backend.py module. Two immediate things bugsare fixed, and after these burnin passes again…
Signed-off-by: Iustin Pop <iustin@google.com>...
Introduce constants.DTS_MAY_ADOPT
DTS_MAY_ADOPT include disk templates that may use disk adoption and will beused in all respective checks.
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Convert TestDelay repeat to the type system
Turns out with the current implementation of the type system a variablecan both be required and not required. Interesting (but perhaps worth afurther look).
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Rename some constants and relax pylint rules
For the type system, we want a slightly relaxed rule for constantnaming, so we update the pylint rule. But the old _TPInt and _TNEStringwere not clear enough, so we expand them.
Introduce a micro type system for opcodes
Currently, we have one structual validation for opcode attributes: the_OP_REQP, which checks that a given attribute is not 'None', and therest of the checks are done at runtime. This means our type system hastwo types: None versus Not-None....
LU.CheckPrereq: do not require implementation
Currently, the base class LogicalUnit's CheckPrereq will raiseNotImplementedError, which means that the child LUs have to implementit. However, many LUs don't actually have a need for this function(hence the many "pass" statements as the only body)....
Some more CheckPrereq/CheckArguments cleanup
For a few LUs, a few tests in, or even the whole CheckPrereq, can bemoved to CheckArguments, as they don't touch state and only do a 'type'validation.
Add OS verification support to cluster verify
For this, we needed to extend the NodeImage class with a few extravariables, and we do a trick in the node verification where we pick thefirst node that returned valid OS data as the reference node, and then...
Move opcode attribute defaults to data structures
LUExportInstance had two opcode fields set to default via both_CheckBooleanOpField and getattr(…, False).
Cleanup LU.ExpandNames versus CheckArguments
When LogicalUnit.CheckArguments was introduced, not all code dealingwith static argument checking was moved to it; many of these checks wereleft in ExpandNames. With time, most of them migrated, and this patch...
Abstract export mode validity check
The export mode is checked in two places with the exact same code…
RAPI: switch evacuate node to the new model
This patch removes the last use of the old-style OpEvacuateNode. It alsofixes the dry-run mode for this RAPI resource - the dry-run parameterwas not used at all before.
Remove the obsolete EvacuateNode OpCode/LU
All code has been switched to the new-style LU… time for cleanup.
Add OS parameters to cluster and instance objects
The patch also modifies the instance RPC calls to fill the osparameterscorrectly with the cluster defaults, and exports the OS parameters inthe instance/OS environment.
Show OS parameters in cluster/instance info
Add support for OS parameters during instance add
This is not yet complete, as it lacks proper support for instanceimport.
_GetUpdatedParams: enhance value removal options
This patch adds controls for whether we recognizeconstants.VALUE_DEFAULT or not as a default value, and also addsdash-prefixes as another way for parameter removal.
Add support for modifying cluster OS parameters
We use _GetUpdatedParams in order to support removal too, and thenvalidate the OS parameters if the OS exists.
Add support for modifying instance OS parameters
We move the instance OS rename checks earlier, as we need to run thevalidation against the new OS, if it has changed.
Add support for OS parameters during import/export
Nothing special here, just copy/adjust the beparams code.
LUDiagnoseOS: add more fields, cleanup
This patch exports all the way from backend a new field ‘api_version’which holds the list of support API versions, and exposes the (alreadycomputed) ‘parameters’ field.
The patch also reworks (again) the field calculation in its Exec()...
Silence a pylint warning
The OS parameters code will bump the number of lines over 10K, and thuswe need to silence this (no, we don't want any other module to becomethis big…, so we use a targeted silence only).
Add os api v20 and related fields to the OS object
Add reading of OS parameters from disk
The patch also modifies the internal methods in LUDiagnoseOS and gnt-osto deal with the format change of call_os_diagnose.
Introduce an RPC call for OS parameters validation
While we only support the 'parameters' check today, the RPC call isgeneric enough that will be able to support other checks in the future.The backend function will both validate the parameters list (so as to...
Remove job object condition
We don't need it anymore, since nobody waits on it.
Parallelize WaitForJobChanges
As for QueryJobs we rely on file updates rather than conditionnotification to acquire job changes. In order to do that we use thepyinotify module to watch files. This might make the client a bit slower(pending planned improvements, such as subscription-based...
Update the job file on feedback
This is needed to convert waitforjobchanges to use inotify and theon-disk version and decouple it from the job queue lock. No replicationto remote nodes is done, to keep the operation fast.
Don't lock on QueryJobs, by using the disk version
We move from querying the in-memory version to loading all jobs from thedisk. Since the jobs are written/deleted on disk in an atomic manner, wedon't need to lock at all. Also, since we're just looking at the...
Add JobQueue.SafeLoadJobFromDisk
This will be used to read a job file without having to deal withexceptions from _LoadJobFromDisk.
jqueue._LoadJobFromDisk: remove safety archival
Currently _LoadJobFromDisk archives job files it finds corrupted. Sincewe want to use it to load files without holding locks, this could causea conflict: we just move the feature to _LoadJobUnlocked which is always...
Add repetition count to the TestDelay opcode
If the repetition count is not passed or is passed as 0 we sleep exactlyone time, otherwise we sleep "repeat" times and log in between.
Merge branch 'devel-2.1'
Add "adopt" to the allowed disk parameters
"adopt" was missing from bd061c3, thus breaking disk adoption.
Fix warnings with Python 2.6
'format' is a new built-in function, and 'bytes' is a new builtin type.We rename this to make pylint happy (and remove potential bugs).
Fix a small bug introduced in cf26a87a
Commit cf26a87a added a tiny typo, which would break non-FQDN argumentsto modify node storage.
Fix the type of 'valid' attribute in LUDiagnoseOS
The update of the valid status in LUDiagnoseOS says:
valid = valid and osl and osl[0][1]
However, in Python, “True and []” (which '[]' we get for an invalid OS)will result in “[]”, and thus the valid field for an OS will be either...
Merge branch 'stable-2.1'
Fix handling of errors from socket.gethostbyname
Socket functions can raise more than just gaierror. Most of the times,socket.gethostbyname_ex will return gaierror, but rarely it will alsoraise herror. For completeness, we catch all socket exceptions with data...
Workerpool.AddManyTasks: check tasks type
Each task has to be a sequence, or the RunTask call will fail.
jqueue.AddManyJobs: use AddManyTasks
Rather than adding the jobs to the worker pool one at a time, we addthem all together, which is slightly faster, and ensures they don't getstarted while we loop.
RAPI client: Add support for Python 2.6
The httplib module used by urllib2 requires its sockets to have amakefile() method to provide a file-like interface (or ratherfile-in-Python-like) to the socket. PyOpenSSL doesn't implementmakefile() as the semantics require files to call dup(2) on the...
Bump RPC protocol version to 40
Many RPC calls have changed in Ganeti 2.2, hence bumping the RPC protocolversion.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix parameter names in SimpleFillBE/NIC docstrings
WorkerPool.AddManyTasks
Useful if we want to add many tasks at once, without contention with theprevious one we added starting.
AsyncAwaker: use shutdown on the socketpair
This makes sure the out_socket can only be used for writing, and thein_socket for reading.
jqueue: make replication on job update optional
Sometimes it's useful to write to the local filesystem, but immediatereplication to all master candidates is not needed.
The _WriteAndReplicateFileUnlocked function gets renamed to_UpdateJobQueueFile, as calling "write and replicate, but don't...
s/queue._GetJobInfoUnlocked/job.GetInfo/
The job queue currently has a static _GetJobInfoUnlocked method.Changing it to be a normal method of _QueuedJob, which makes more sense.
Abstract loading job file from disk
Move the work from _LoadJobUnlocked to _LoadJobFileFromDisk, which canthen be used in other contexts as well. Also, if we fail to deserializethe job, archive it as well (before we archived it only if we failed tocreate the related object, but kept it there if deserialization failed....
jqueue: simplify removal from _nodes
Somewhere we do try/del/except and somewhere just pop. Using popeverywhere saves lines of code.
ListVisibleFiles: do not sort output
Among all users, turns out just one may need the output to be sorted.All the others can cope without.
Remove a TODO
Since OS objects are not stored in the configuration, we cannot putos_hvp there, therefore the TODO is obsolete…
Rework LUSetInstanceParams._GetUpdatedParams
Currently, this function does three things:- special handling of constants.VALUE_DEFAULT- type enforcing of the resulting dict- filling the dictionary with defaults
However, except for the first one, the second two do not belong in this...
Split the core-OS and instance-specific env
Since we'll need to be able to generate the OS-specific environmentseparately from the instance one, we move it to a separate function. Wealso add a new OS_NAME env. var which is identical to the INSTANCE_OSone (which won't exist for OS-only environments)....
Add cluster.SimpleFill*() functions
Currently, the existing cluster.Fill* functions take as argument aninstance. This means that in any case where we don't have an actualinstance object, we have to resort to calling the low-levelobjects.FillDict function....
Merge branch 'devel-2.1' into master
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>
Fix a bug in instance startup with custom hvparams
Since the introduction of OS-specific hvparams, we shouldn't ever useobjects.FillDict directly for instances, but instead go via the clusterobject. Otherwise the os_hvp will be ignored.
Fix unsafe variant initializer in _TryOSFromDisk
In case an OS has inconsistent declarations, we might get into a casewhere one node reports a valid variants list (with OS API >=15), andanother node has OS API < 15, in which case its supported_variants gets...
Disallow DES for SSL connections
Older OpenSSL versions include DES-CBC3-* ciphers when specifying theHIGH group of ciphers. Removing potentially weak ciphers from the listof allowed ciphers ensures only strong ciphers are considered for SSLconnections....
Start instance after creating snapshots for export
This restores functionality lost in commit 387794f8. Found duringtests using QA scripts. An instance should be started after ithas been temporarily shutdown for an export.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Use import/export magic for backup/import and inter-cluster moves
This should prevent bugs in our code from accidentally overwritingdisks.
Disable compression for all intra-cluster imports/exports
Tests have shown that usually we're CPU-bound for intra-clusterimports/exports. Disabling compression will help with this.
Some versions of OpenSSL, depending on the build options, alsocompress transparently. This will need further work in Ganeti....
backend: Add support for import/export magic
import/export daemon: Add support for a magic prefix
This “magic” value will be used to ensure that we don't accidentiallyconnect to the wrong daemon (e.g. due to a bug), comparable to DRBD'sper-disk secret. Just depending on the SSL certificate isn't enough...
import/export daemon: Simplify command building
Instead of appending strings, stage parts in a list. Building the "dd" command is moved to a separate function.
import/export: Limit max length of socat options
import/export: Validate remote host/port
The hostname and port received from the remote cluster shouldbe validated, just in case.
utils: Add function to validate service name
Handle ESRCH when sending signals
Upon sending signals, ESRCH can be reported when the target nolonger exists.
Fix RpcResult.Raise error code
A typo in the Raise() method of rpc.RpcResult means that any remoteerrors will lack an appropriate error code; this will confuse e.g. RAPIusers.
Cache a few bits of status in jqueue
Currently each time we submit a job we check the job queue size, and thedrained file. With this change we keep these pieces of information inmemory and don't read them from the filesystem each time.
Significant changes include:...
ListVisibleFiles: do optional sorting
Fix a TODO in _QueuedJob
Rather than raising Exception use GenericError and explain a bit betterwhat happened.
Remove unused parameter from function
This also removes the relevant pylint disable.No point in keeping unused parameters around: if/when we need them it'seasy to add it back.
Optimize _GetJobIDsUnlocked
Currently we sort the list of job queue files twice (once inutils.ListVisibleFiles with sort and then later with NiceSort). We applythe _RE_JOB_FILE regular expression twice (once in _ListJobFiles andonce in _ExtractJobID). This simplifies the code a little, and a couple...
jqueue: Rename _queue_lock to _queue_filelock
The name clarifies the difference between this and the internal lock.Also explain a bit better what it is.
jstore._ReadNumericFile: use utils.ReadFile
Enforce consistency in disks and nics input dicts
With this change unknown disk and nic parameters will be refused, ratherthan silently ignored, so that one can't pass them in by mistake and notrealize what went wrong.
TLMigrateInstance: pass lu to _Check*
The various _Check* helper functions expect an lu to be passed in, butthe TL is passed instead. This works... sometimes! :)
Remove locking._CountingCondition
This class is unused and untested. We must have forgot it around.
Remove the job queue drain rpc call
This call was introduced but never used. In two years.Since it's just creating/removing a file it can also be in simpler ways,without a special rpc call, if/when we need it again. In the meantime,let's give it to history....
Move fake hypervisor run dir under ganeti
This makes it uniform with the other hypervisors.
_BaseCondition: allow saving/restoring state
SharedLock _acquire_restore and _release_save
If a shared lock is used inside a condition, we need to make sure thatit's reacquired in the same way as it was originally, after the wait.
Submit[*each*]Pending job
This is useful so we can test both SubmitJob and SubmitManyJobs.
cfgupgrade: Local variable for cluster-domain-secret filename
This is necessary to allow cfgupgrade to work on a non-standard directory.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Show formatted ETA for disk sync and import/export