Add prealloc_wipe_disks as a cluster-wide configuration variable
This is the first step for the support of wiping block devices priorto creation of the instance.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge branch 'devel-2.2'
Conflicts: lib/rpc.py (trivial, copyright header)
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
RPC: disable curl's Expect header
This patch solves the very slow (~8-9 seconds) gnt-instance modifybehaviour. Well, it solves in general the slow RPC behaviour, but it wasmost visible in that LU.
It seems that curl's behaviour with regard to file uploads (via PUT) and...
jqueue, CancelJob: Check status only once per call
This simplifies the code a bit--the status is only checked once.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix a rare bug in StartDaemonChild and GenericMain
I've seen cases where the result from str(sys.exc_info()[1]) is ""; thisbreaks the error reporting as the parent relies on non-empty errormessages to properly detect child status (otherwise it will try to read...
Enhance the error reporting
Since daemon startup error will be often related to socket errors, so itmakes sense to change the original reporting:
Error when starting daemon process: "(98, 'Address already in use')"
Into:
Error when starting daemon process: 'Socket-related error: Address...
Change daemon.GenericMain/utils.Daemonize workflow
This patch copies the pipe-based error reporting functionality fromutils.StartDaemon (I gave up for now on tryin to merge the two).
This patch will fix two longstanding bugs:
- if we fork, we lose all error reporting from the child to the original...
Change utils.GenericMain protocol
Currently, GenericMain does a two-staged workflow:
- Check, before forking- then Exec, after forking
This means we don't have any possibility to treat preparation work(before the daemon is ready for work) different from the actual work....
Use only one version of WritePidFile
This patch merges the pid file handling used for ganeti-* daemons andimpexp daemons. The latter version is used, since it's more reliable:uses locked pid files as opposed to checking 'live' processes.
Signed-off-by: Iustin Pop <iustin@google.com>...
Abstract daemon file descriptor setup
This does some slight changes:
- Daemonize() doesn't explicitly close the file-descriptors anymore, but only implicitly via the usage of dup2- StartDaemonChild uses separate devnull for stdin (rdonly) and stdout/stderr (wronly), or if using a log file, it uses it in append...
Abstract some daemon functionality
This patch abstracts the chdir/umask/setsid functionality, which isidentical in the code functions, just that Daemonize did the chdir/umaskin the second child; with this change it does it in the first, asStartDaemon....
Export VG name via LUQueryConfigValues
This will be used by LUXI client programs to display the VG name.
LUGetTags: Acquire locks in shared mode
Retrieving tags can be done while the lock is shared. Only writingneeds to be exclusive.
Also add a FIXME for cluster tags, where the code currently doesn'tuse any locks except the config lock.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
LUDelTags: Improve formatting of error message
Use utils.CommaJoin to add spaces after comma, clean up code a bit.
Before: Tag(s) 'bar','baz','foo','moo' not foundAfter: Tag(s) 'bar', 'baz', 'foo', 'moo' not found
cli: Move parsing of --net option to separate function
This function will also be used in tools/move-instance.
kvm: collapse two consecutive extend calls
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
kvm: Introduce support for -mem-path
Using hugepages, KVM instances can get a good performance boost. Toactivate that, we need to pass the -mem-path argument to KVM along withthe mount point of the hugetlbfs file system on the node.
For the sake of memory availability computation, we use the -mem-prealloc...
Conflicts: lib/objects.py (trivial, strange that this one, and only this one, conflicted)
Rename the _oss cluster vars to _os
Per the mailing list discussion, rename _oss to _os, both in cluster parametersand in the rest of the code.
This is just an s/_oss/_os, with the exception of a small bit of cleanuparound the helper_os function in cmdlib.py....
KVM: Add function to check the hypervisor version
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix instance rename regression from 3fe11ba3
Committ 3fe11ba3 broke the instance rename as we don't use the FQDNanymore. This fixes it.
Sort OS names and variants in LUDiagnoseOS
The OS list and variants as returned from LUDiagnoseOS is not sorted,and gnt-instance reinstall doesn't sort it either. This means that itthe menu that users are present with is inconsistent across clusters,and that is confusing....
Change behaviour of OpDiagnoseOS w.r.t. 'valid'
This patch changes the behaviour of OpDiagnoseOS with regards to the'valid' field to be similar to the one for the hidden/blacklistedfields: unless this field is requested, invalid OSes are filtered out....
Allow gnt-os modify to change the new OS params
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add two more _T-type tests
These are useful for more in-depth checking of some kinds of arguments.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Add blacklisted/hidden OS support in LUDiagnoseOS
This changes the behaviour of LUDiagnoseOS significantly.
The addition of hidden/blacklisted OSes would mean that each user-facingclient would have to filter intentionally such OSes from display, whichis not a good choice. Rather, the patch makes LUDiagnoseOS not return...
Restrict blacklisted OSes in instance installation
Add two new cluster settings
The new variables are:
- a list of hidden OSes, that should not be displayed to the users in interactive selection (e.g. reinstall); however, if they are selected, they can be used- a list of OSes that should be hidden and blocked from install-time selection...
Abstract OS name/variant functions
Currently, the computation of the 'pure' name or the variant ishardcoded and spread around the functions that need it. This is notnice, and in the future we'd spread it even more with more usage ofvariants/pure os names....
Avoid nodegroup name/uuid conflicts
Forbid nodegroups to be called with a name that matches the UUID regularexpression. Uppercase versions are forbidden as well.
Move the uuid regexp to utils.py
Fix docstring typo in jqueue._JobProcessor._MarkWaitlock
epydoc complained:“File …/ganeti/jqueue.py, line 886, inganeti.jqueue._JobProcessor._MarkWaitlock Warning: Redefinition of type for job”
jqueue: Use priority for acquiring locks
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
mcpu: Implement priority for lock acquiring
Until now the priority for lock acquires couldn't be passedwhen running opcodes.
locking: Implement priority in Ganeti lock manager
locking: Don't set default priority as keyword default
This allows users of these classes to simply pass None if they want to use thedefault value (the actual default is an internal constant), instead ofdynamically constructing the keyword arguments.
jqueue: Use timeout when acquiring locks
As already noted in the design document, an opcode's priority isincreased when the lock(s) can't be acquired within a certain amount oftime, except at the highest priority, where in such a case a blockingacquire is used....
Migrate call from backend._GetVGInfo to bdev.LogicalVolume.GetVGInfo
This patch removes duplicate code found in backend which also needs toget VG infos. To make it simpler we moved to bdev.LogicalVolume.GetVGInfo.
Signed-off-by: René Nussbaumer <rn@google.com>...
jqueue: Introduce per-opcode context object
This is better to group per-opcode data.
mcpu: Adjust lock acquire strategy
The changes to job queue processing require some changes on this class'interface. LockAttemptTimeoutStrategy might move to another place, but that'llbe done in a later patch.
mcpu.Processor: Raise exception on lock acquire timeout
Right now the timeout is not passed by any caller, making the codeeffectively go back to blocking acquires. Since the timeout is alwaysNone, no caller needs to be changed in this patch.
This change also means that any LUXI query handled by ganeti-masterd...
jqueue: Rename current_op to better reflect what it actually is
jqueue: Separate function for in-memory variables
Merge branch 'stable-2.2' into devel-2.2
jqueue: Add unittest for _QueuedJob.CalcStatus
Use free space in vg instead of biggest free pv space for a snapshot
Even for snapshot we looked at the biggest free pv space even thoughthe vg might have fit the snapshot we aborted if one of the pvs was toosmall. This patch fixes this by looking at the vg size instead of the pv...
Merge branch 'devel-2.1' into devel-2.2
Fix mac checker regex
Currently, the mac checker regex could match a corner case of11:22:33:44:55:66: (one extra colon at the end). We fix this, and wealso move the regex compilation outside of this function, at modulelevel.
Ignore failures while shutting down instances during failover from offline node
Don't abort failover if instance shutdown doesn't work on a node markedoffline. The node is offline, so the instances living on it are too. Beforeyou had to use --ignore-consistency to archieve that....
cli: Expose priority option and pass priority to master daemon
jqueue: Change model from per-job to per-opcode processing
In order to support priorities, the processing of jobs needs to bechanged. Instead of processing jobs as a whole, the code is changed toprocess one opcode at a time and then return to the queue. See the...
jqueue: Use priority for worker pool
A small helper function is added to make this easier. Priorities are notyet used in all necessary places.
Fix migration on new KVMs
New KVMs (0.12.1.2-el6 and 0.13.5 tested) exit immediately afterunsuccessful network connection when they are in "-incoming" mode. Thesimple check netutils.TcpPing causes remote kvm to exit so the migrationwill always fail. This check is also redundant by the way as if the...
cli: Pass options in {Add,Remove}Tags
They'll be used for job priorities. Also add an empty line tognt-os where it's missing.
jqueue: Add missing docstring to _QueuedJob.Cancel
This was forgotten in commit 099b2870b.
Bail out if daemon gets fired up under wrong uid
This patch bails out in early stage, if the user invoking the daemondoesn't match the user set at configure time.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
opcode summary: improve display for list summaries
Currently, opcodes like NODE_EVAC_STRATEGY look bad:
89684 error NODE_EVAC_STRATEGY([u'node3'])
With this patch, we try to render list arguments a little bit better:
89684 error NODE_EVAC_STRATEGY(node3)...
workerpool: Fix typo
A call to logging.debug was missing an argument, leading to complaintson stderr at runtime.
jqueue: Move CancelJob logic to separate function
Moving the internals of this function will allow it to be used fromunittests in the future. Splitting this into a pure, side-effect freefunction and an impure one makes the pure function easily testable....
(no conflicts, took LGTM from original commit)
cmdlib: Fix type of “name” parameter for tag operations
The parameter “name” is be None for cluster tags.
rlib2: Set tag operation param “name” to None for cluster tags
Otherwise parameter verification in the master daemon fails.
Stop all daemons precautiosly before trying to start ganeti-noded again
Please note that if the pid file is broken or missing we'll not catchthe process (if any is running) and it's up to the user to fix this state
Check for duplicate nodegroup names
Since the nodegroups dict is indexed by uuid duplicate names mighthappen as a result of bugs. Add a check to prevent them.
Split a long docstring line in objects.py
InitConfig: create nodegroups as well
This patch also ensures that the initial configuration has all theneeded UUIDs and that they are unique, by using aTemporaryReservationManager inside InitConfit to generate them.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Export nodegroups list (names/uuids) via ssconf
Add nodegroup bash autocomplation
We autocomplete both by nodegroup name and uuid.
Add nodegroup option to AddNode
Add node's nodegroup field
If a node doesn't have a node group we'll upgrade the config making itthe cluster default. Also the node add and removal operations arechanged to set/clear the node group correctly. Finally we populate the"members" list of nodegroups on config load with the value from the...
Check for nodegroup uuid indexing
Since the uuid is immutable the probability of it getting out of syncbetween the object and the dict key is very low. Still, checking ischeap, so we do it to be more sure nothing is wrong.
config.LookupNodeGroup
This function allows a node group to be looked up by name or uuid.If no nodegroup is specified and only one exists, that one is returned.
ConfigWriter: create the default node group
If no node groups exist we'll create a new default empty one.
Add a 'nodegroups' slot to ConfigData
Also: - reformat the slots declaration of ConfigData - call UpgradeConfig on each node groups to handle future upgrades - add nodegroups special case in {To,From}Dict - add nodegroups to _AllUUIDObjects
_ContainerFromDicts: handle None source
When _ContainerFromDicts is called on some element which doesn't existin the config, because it is yet to be upgraded, it will receive itsvalue as None. We take care of this case by using an empty element ofthe required target type....
Add a new NodeGroup config object
The "members" slot of this object is not serialized, and is discardedwhen deserializing, initializing it explicitly to [].
cli: Add option definition for priority
jqueue: Ensure only accepted priorities are allowed for submitting jobs
Quoting the design document: “Submitted opcodes can have one of the prioritieslisted below. Other priorities are reserved for internal use”. Submitting jobsat priority -20 should not be allowed....
Add support for job priority to opcodes and job queue objects
This allows clients to submit opcodes with a priority. Except for beingtracked by the job queue, it is not yet used by any code.
Unittests for jqueue._QueuedOpCode and jqueue._QueuedJob are provided for...
Add job priority constants
Remove mcpu's ReportLocks callback
This is no longer needed with the new lock monitor. One callback is kept tocheck for cancelled jobs.
Revert "jqueue: Resume jobs from “waitlock” status"
This reverts commit 4008c8edae31a3971fa8c4b200238afc8005d3d4.
While it worked in my initial tests, I've now found cases where this doesn'twork properly as it is. More work is needed and will be done as part of the...
Fix OS_VARIANT variable setting
This was introduced in efaa9b06d1e1e6d1678d0edd75b1ba37cf0de3d9.
in OSCoreEnv: inst_os.name is pure operating system name (without variant) as variant is stripped in OSFromDisk(). So we always get variant = inst_os.supported_variants[0] (first...
Fix pylint warning in http/__init__.py
My bad for not seeing this before:R0201:614:HttpBase.GetSslCiphers: Method could be a function
Allow SSL ciphers to be overridden in HTTP server
Users of this class, such as the RAPI server, might want to override or adjustthe default SSL cipher defined in a constant.
jqueue: Resume jobs from “waitlock” status
After an unclean restart of ganeti-masterd, jobs in the “waitlock” status canbe safely restarted. They hadn't modified the cluster yet.
jqueue: Move queue inspection into separate function
This makes the init function a lot smaller while not changingfunctionality.
jqueue: Don't update file in MarkUnfinishedOps
This reduced the number of updates to the job files. It's used in two placeswhile processing a job and the file is updated just afterwards.
locking.SharedLock: Update class docstring
This was already outdated when the initial version of SharedLock was addedin commit 162c1c1f1 (February 2008).
locking: Implement priorities in SharedLock and LockSet
For proper support of job priorities, jobs' locks need to respectpriorities. Otherwise it could happen that a job with a lower prioritycould get a lock before a job with a higher priority (depending on...
Remove utils.EnsureDir as this is done by ensure-dirs.in now
The config now should also belong to confd group and readable by it
Move job queue to new ganeti.runtime