Introduce git reference/tag tracking for debugging
This patch adds a new vcs-version file that is generated via git (andcan be adapted if VCS is changed) and then embebbded as VCS_VERSION inthe constants module.
This means two things:- local modifications without committing to git (or when using a tar.gz...
Fix epydoc warning "Lists must be indented."
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Convert RPC client to PycURL
Instead of using our custom HTTP client, using PycURL's multiinterface allows us to get rid of the HTTP client threadpool.The majority of the code is still in the ganeti.http.clientmodule.
A simple per-thread HTTP client pool gives cURL a chance to...
Implement lock names for debugging purposes
This patch adds lock names to SharedLocks and LockSets, that can be usedlater for displaying the actual locks being held/used in places where weonly have the lock, and not the entire context of the locking operation....
Fix pylint complaints introduced in commit e58f87a958c
Due to a small mistake I missed three non-critical pylint complaints forcommit e58f87a958c. They're fixed with this patch.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Signed-off-by: Iustin Pop <iustin@google.com>...
LXC: Add cpu_mask hypervisor parameter
Also implement syntax checking.
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add ParseCpuMask() utility function
Also adds a generic ParseError exception.
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add a migration type global hypervisor parameter
Since migration live/non-live is more stable (e.g.) for Xen-PVM versusXen-HVM, we introduce a new parameter for what mode we should use bydefault (if not overridden by the user, in the opcode).
The meaning of the opcode 'live' field changes from boolean to either...
Add test for some aspects of job queue
This new opcode and gnt-debug sub-command test some aspects of thejob queue, including the status of a job. The bug fixed in commit2034c70d507 was identified using this test. A future patch willrun this test automatically from the QA scripts....
LUVerifyCluster: update _ValidateNode description
Change _ValidateNode description to reflect what the function actually does.
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
KVM hypervisor: Use utils.ShellWriter for network script
This patch converts hv_kvm to use utils.ShellWriter for writingthe network script. It also adds a few unittests (the firstfor any hypervisor modules).
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Move ShellWriter class to utils
Also add unittest.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
jqueue: Factorize code waiting for job changes
By splitting the _WaitForJobChangesHelper class into multiple smallerclasses, we gain in several places:
- Simpler code, less interaction between functions and variables- Easy to unittest (close to 100% coverage)...
Merge remote branch 'origin/devel-2.1'
Conflicts: test/ganeti.rapi.client_unittest.py: Trivial test/ganeti.rapi.rlib2_unittest.py: Trivial...
RAPI client: Implement old instance creation request format
Commit 8a47b4478 implemented instance creation in the RAPI client,but it left out support for the old instance creation request format.This patch now implements the old format as good as possible. This...
rlib2: Use constants for disk and NIC parameters
These constants were added in commit bd061c35, but the parsing codewas not updated. This also fixes a bug where a NIC's MAC addresswasn't used.
Use reserved documentation IPs and domains
Use RFC 5737 IP addresses and RFC 2606 domain names in allunittests, docs, qa and docstrings.
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Provide feedback function for all LU methods
By exposing mcpu's _Feedback function (now renamed to “Log”) to LU's,methods like ExpandNames can also write to the job execution log.
jqueue: Setup inotify before checking for any job changes
Since the code waiting for job changes was modified to use inotify,a race condition between checking for changes the first time andsetting up inotify occurs. If the job is modified after the check...
cli.SubmitOpCode: Support custom job reporter
This is necessary to reuse SubmitOpCode while adding processing forcustom message types.
Add function to format all job log messages
Just calling utils.SafeEncode on the log message failed when itwasn't of the type ELOG_MESSAGE and not a string. Now non-messagelog entries are formatted using repr().
baserlib: Fix feedback function
The feedback function is called with only one parameter, a tuplewith the message details.
Confd IPv6 support
This patch series basically adds a new parameter 'family' to the constructorsof daemon.AsyncUDPSocket and confd.client.ConfdUDPClient. This enables theusers of these two classes to support IPv6.
In ganeti-confd.ConfdAsyncUDPClient a method to check the address families of...
LXC: Create per-instance log files
This replaces the single global log file with per-instance logs.The instance log file is not truncated when the instance is started.
LXC: Fix wording of error messages
LXC: Fix GetInstanceInfo()
Don't try to get cgroups info if instance is not running.
Introduce lib/netutils.py
This patch moves network utility functions to a dedicated module.
LXC: Fix GetAllInstancesInfo()
Add oper_vcpus instance status field
This introduces a new instance status field, named "oper_vcpus".It contains the actual number of VCPUs an instance is using asseen by the hypervisor.
LUCreateInstance: use cluster-wide iallocator
LUCreateInstance uses the cluster-wide default iallocator if no iallocator orprimary node is specified manually.
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>...
LUNodeEvacuationStrategy: Use default iallocator
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add default_iallocator cluster parameter
Add a cluster parameter to hold the iallocator that will be used by defaultwhen required and no alternative (manually-specified iallocator ormanually-specified node(s)) is given.
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>...
Add _CheckIAllocatorOrNode for common iallocator/node checks
_CheckIAllocatorOrNode will be called by LUs wishing to use an instanceallocator or a target node. It performs sanity checks and will modify the LU'sopcode's iallocator slot to use the cluster-wide allocator if...
LXC: Report actual number of CPUs
Merge branch 'devel-2.1'
Mlockall: decrease warnings if ctypes module is not present
Node daemon prints a lot of warnings if --no-mlock option is not specified andctypes module is not present.
With the following patch the warning is printed only at noded startup.
Signed-off-by: Luca Bigliardi <shammash@google.com>...
Add a delay in master failover
I have seen some very seldom errors where (it seems) the address isstill live for a short while after removing it from the old master, thusthe new master will fail in startup/adding its own IP address.
To prevent against this, we add a delay/retry before we proceed, if the...
LXC: add lxc.console to the generated lxc.conf file
LXC: Use lxc-info to get instance info
Fix opcode transition from WAITLOCK to RUNNING
With the recent changes in the job queue, an old bug surfaced: we neverserialized the status change when in NotifyStart, thus a crash of themaster would have left the job queue oblivious to the fact that the job...
cli.JobExecutor.WaitOrShow: always return status
Currently, for the 'wait' case, we return a list of tuples (status,result), in the order of submitted jobs, but we don't return anythingfor the no-wait case.
This patch changes the no-wait case to return a list of tuples (status,...
Rework the export failure handling
Currently, the way to signal export failures is by the return value.This means that if a client doesn't check the values (e.g. burnin), anyfailure is being ignore. And this is what we've been doing forever inburning (not actually testing that the export is successful)....
Add utils.GetMounts()
LXC: use utils.GetMounts()
hv_chroot: use utils.GetMounts()
Add drbd helper and storage options
Report drbd helper in query info LU
Check and set drbd helper in set params LU
Check and set drbd helper during bootstrap
Add drbd_helper rpc call
Check drbd usermode helper in cluster verify
Set drbd usermode helper on config upgrade
Generalize a recursive check on logical disks
Add drbd_usermode_helper to configuration
VerifyNode: add usermode helper reply
BaseDRBD: provide a way to query usermode_helper parameter
OpCreateInstance: do not require hv/be/os params
It is perfectly legal to create an instance using only defaults(although beparams will be most times passed in), so let's relax therequirement for these three parameters.
Signed-off-by: Iustin Pop <iustin@google.com>...
A few more type definitions
This is to simplify the type declarations in the actual LUs.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Make _CheckDiskTemplate a valid type checker
Rework the "type" system
This patch merges the _OP_REQP and _OP_DEFS class attributes into a_OP_PARAMS list, which holds both. The associated unittest checks thatall opcode attributes are declared and checked, and that no LU uses theold fields (could be removed later)....
Remove _CheckBooleanOpField
This is no longer used, and we can remove it.
Merge branch 'stable-2.1'
RapiClient: fix multi-authentication in Python 2.6
In Python 2.6 the urllib2.HTTPBasicAuthHandler has a "retried" count forfailed authentications. The handler fails after 5 of them. To solve thiswe reset the handler's "retried" member variable to 0 after every...
utils.OwnIpAddress: Change try/except for Python 2.4
Python 2.4 doesn't support “except” and “finally” in the same block.
RAPI client: Switch to pycURL
Currently the RAPI client uses the urllib2 and httplib modules fromPython's standard library. They're used with pyOpenSSL in a very fragileway, and there are known issues when receiving large responses from a RAPIserver....
baserlib: Use boolean type for boolean variables
This does not yet fix all issues in the RAPI interface which wereintroduced with the type system. More testing is needed.
Fix check in gnt-instance modify -t
Currently when changing the instance disks the instance status (whichmust be down) is only checked if the target status is among the mirroredones. This is incorrect: the instance must be down either way, even ifthe desired final status is non-mirrored....
Adding check_name option to the opcode and luxi call for instance rename
This will allow instance rename without dns check as it does for instanceadd.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Enable utils.OwnIpAddress to work with IPv6 addresses
Currently a TcpPing using the target address as source is used todetermine wheter or not an IP address belongs to a node. This does notwork for IPv6 anymore (I'm not sure why) as binding to IPv6 localhost as...
Barebones LXC hypervisor
This needs lots of work, but it can successfully launch an LXC-basedinstance. See the docstring for the limitations/work to be done.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>
DRBD IPv6 support
Support IPv6 configuration for 'drbdsetup show' parser and add unittestsconcerning that case. Renames some data files to use consistent namesclarifying their usage.
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
IPv6 support for utils.TcpPing()
Add function to retrieve family of an ip address
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Introduce utils.IsValidIP{4,6}()
This patch introduces functions to check for valid IPv4 and IPv6addresses and converts IsValidIP() to return True if it is either a IPv4or a IPv6 address.
For now we do not change the functional behavior and replace IsValidIP...
Fix osparams bug when gnt-cluster modify'ing
Commit 625ac1135834c27a2e78fd61bf2df2ce29ad2966 introduced 'osparams'slot in OPSetClusterParams which was not passed when invoked viagnt-cluster modify.
Signed-off-by: Manuel Franceschini <livewire@google.com>...
Introduce IPv6 constants
Replace '0.0.0.0' with constant
Rename some constants to facilitate IPv6 support
Pass force variant option at instance creation
This was supposed to be done in"06073e857e3b518c7195d57306ae01793240c0c2" but by mistake the field wasadded to batch create rather than the normal instance create.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Fix a few uncommon pylint errors in compat.py
- It's ok if the optional modules functools, roman and hashlib are not there.- It's ok to reference any and all before defining them. We're doing all this exactly because, under python 2.4, they are undefined....
AsyncTerminatedMessageStream: send_message
This function adds the ability for a AsyncTerminatedMessageStream tohave a thread-safe message delivery function.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
AsyncTerminatedMessageStream: limit message count
Currently the message stream can process any number of messages inparallel (if they get dispatched to different threads or processes).In order to limit their number we only handle messages and read from...
Better specify old object slots removal policy
The comments seem to imply we can remove deprecated slots after 2.1 isreleased. This is not quite true, if we still want to support upgradingstraight from 2.0. So let's specify better that they can be removed only...
Fix a type declaration error in LUTestAllocator
Also, improve logging of type errors, to ease debugging.
Fix _UpgradeConfig docstring
jqueue: remove the _big_jqueue_lock module global
By using ssynchronized in the new way, we can remove the module-global_big_jqueue_lock and revert back to an internal _lock inside the jqueue.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
ssynchronized: act on a class member
The ssynchronized decorator takes the lock to act on in input.With this change we allow a string to be passed, and if so we assume thefunction it protects is a class method, and we act on the member of theclass itself named as the string we got....
Share the jqueue lock on job-local changes
We can share the jqueue lock when we do per-job updates. These onlyconflict with updates/checks on the same job from another thread (eg.CancelJob, ArchiveJob, which keep the lock unshared, since they are lessfrequent)....
_OpExecCallbacks abstract _AppendFeedback
Move some code to a decorated function rather than explicitelyacquiring/releasing the lock in AppendFeedback.
Remove utils.LockedMethod
All users of this function were converted to another method, hence wecan remove it.
jqueue: convert to a SharedLock()
Remove the jqueue _lock member and convert to a _big_jqueue_locksharedlock. This allows smooth transition from the old single lock to amore granular approach.
MarkUnfinishedOps: update job file on disk
Every time we call MarkUnfinishedOps we do it in a try/finally blockthat updates the job file. With this patch we move the try/finallyinside. CancelJobUnlocked is removed, because it just becomes a wrapperover MarkUnfinishedOps with two constant values....
Remove spurious empty line
Remove pred from compat.any/all
This makes it compatible with the python builtin, and we can even usethe builtin when running under the right version of python. The all andany functions are renamed to _all and _any, so that they can be tested,and (non)existing unittests are updated (translation: there are no unit...
Two more fixes for OS params and opcode defaults
If the OS is not using API v20, the parameter verification should beentirely skipped.
The second change is a simple typo.
Fix breakage due to OS parameters
I was using wrong python installation path (thanks Guido!), so I was notactually testing the new backend.py module. Two immediate things bugsare fixed, and after these burnin passes again…
Introduce constants.DTS_MAY_ADOPT
DTS_MAY_ADOPT include disk templates that may use disk adoption and will beused in all respective checks.
Convert TestDelay repeat to the type system
Turns out with the current implementation of the type system a variablecan both be required and not required. Interesting (but perhaps worth afurther look).
Rename some constants and relax pylint rules
For the type system, we want a slightly relaxed rule for constantnaming, so we update the pylint rule. But the old _TPInt and _TNEStringwere not clear enough, so we expand them.