(2.12) Convert QmpMessage to a new-style class
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
(2.12) KVM: move monitor & QMP classes to a submodule
We move the HMP and QMP classes to the hv_kvm.monitor module.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
(2.12) Convert hv_kvm to a package
Weighing in at 3k LOC, it is probably time to split hv_kvm into anumber of modules before adding new code.
(2.12) kvm: Add migration capabilities as an hvparam
Latest QEMU versions support various migration capabilities. Eachcan be enabled/disabled with 'migrate_set_capability' monitorcommand.
Version 1.7.0 defines x-rdma-pin-all, auto-converge, zero-blocks,...
(2.12) Add TAP interface name as argument in '_OpenTap'
This is necessary in order to create TAP interfaces for thecommunication mechanism, which are named 'gnt.com.%d', where '%d' is aunique number within a given node.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>...
(2.12) Rename 'CdromOption' to '_CdromOption'
... because it is a protected definition.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
(2.12) Fix KVM cdrom image URL in the second cdrom drive
(2.12) kvm: check that the ISO image is there if it's a URL
Perform a simple urllib2 check on ISO images specified as URL beforeinstance start, so as to work around qemu bug #597575 [1].
[1] https://bugs.launchpad.net/qemu/+bug/597575
Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>...
(2.12) Allow KVM to boot from HTTP
New versions of KVM support booting from HTTP-hosted ISO images, vialibcurl. This patch adds a proper check to allow defining either a sane,absolute path or an HTTP URL as an iso image path.
Remove "format=raw" from the cdrom device options when iso_image starts...
(2.11) Add andRestArguments to IDiskParams
In this way, we cann pass through the opaque parametersrequired for disk creation and modification in the case ofexternal storage.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>...
(2.11) Add function providing the canonical andRestArguments
The field catching the remaining fields will always be of the sameshape, so add a function for this to make usage simple.
(2.11) Add genAndRestArguments :: Gen (Map String JSValue)
So that objects using AndRestArguments are available for testing.As the AndRestArguments are intended for passing through additionalparameters passed on the command line, we restrict them to the...
(2.11) Add additional constructor AndRestArguments to OptionalType
A field of this type will capture all the remaining fieldsof an object as JSValues. Obviously, the intended use isto have precisely one such field. This mechanism will allowto pass opaque values trough, as it is, e.g., required for...
(2.11) Make BlockDev subclasses adhere the interface for Create
In commit 702c3270 two new parameters were added to theCreate function of BlockDev. Make subclasses also adherethis specification.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
(2.11) Make BlockDev subclasses adhere to new interface
In commit 702c3270 two new parameters were added to theconstructor of BlockDev. Make the subclassess accept theseadditional parameters as well.
(2.11) Make disk.name and disk.uuid available in bdev
Until now Disk name and uuid was not available on bdev level.In case of ExtStorage, this info is useful, and may be for othertemplates in the future too.
This patch treats the name and uuid object slots just like the size...
(2.11) Add 'provider' to IDiskParams
IDISK_PROVIDER was included in python's IDISK_PARAMS, so itshould also be included in the Haskell code.
Now that luxid creates and enqueues jobs, without this patch theExtStorage interface is broken as the user can not pass the disk...
(2.11) Fix class methods in KVM hypervisor
Several methods in KVM hypervisor behave like 'classmethods', however,they did not have the '@classmethod' annotation. This patch adds'@classmethod' annotations to several of these methods with the goalof making '_StopInstance' available to the QA. This step is necessary...
(2.8r) Workaround for Issue 621
Upon LUNetworkDisconnect() and LUNetworkConnect() try to acquireall cluster's instances.
By that _LS_ACQUIRE_ALL acquire mode is set and not_LS_ACQUIRE_EXACT and thus the deleted lock does cause any problem.
NOTE: This workaround is not merged upstream. They prefer to have...
Disable CheckFileStoragePathNotWritable test
..because it fails when runs as root.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
kvm: Delegate socket handling to monitor module
Introduce @_ensure_connection decorator that properly handlesthe socket connection (i.e. connect, close) and propagatesany exceptions raise in the decorated method.
Fix python shebang line in tools as well
While most of our top-level python scripts used to be generatedin the Makefile already, the scripts under tools used to be"ready to use". However, in our current situation where thepython 2 interpreter is called different on different platforms...
Do not hard-code python path
...in our automatically generated top-level python files.Instead use the automatically detected python path.
Add a target to inspect make variables
...this will allow inspecting, e.g., the effective compilerflags without actually running the compiler. For example,to see the value of the HFLAGS,
make print-HFLAGS
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Have fatal warnings on in developer mode
Unless you develop Ganeti, it is not a problem if some ofthe functions used are deprecated in you newer haskellenvironment. So allow those builds for normal users.
Add a flag --developer-mode
So that we can have fatal warnings in development, butpeople can still build old Ganeti versions with newercompilers despite some functions are deprecated by now.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Fix check for sphinx-build from python2-sphinx
if using python2-sphinx '/usr/bin/sphinx-build2 --version' prints"Sphinx (sphinx-build2) 1.2.3" which didn't match the regex
Signed-off-by: Neal Oakey <neal.oakey@googlemail.com>Reviewed-by: Helga Velroyen <helgav@google.com>
use the Python interpreter from env
use the Python interpreter which has been set via PYTHON env-var atconfigure time
Signed-off-by: Neal Oakey <neal.oakey@googlemail.com>Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
kvm: Use vhostfds obtained by OpenTap
During instance startup, if vhost_net is True, OpenTap() opens/dev/vhost-net device to obtain some fds. These fds were neverused. This patch adds the vhostsfd option to the --netdev option.
hotplug: Retry VerifyHotplugCommand up to 5 times
It seems that QMP upon device_del returns without QEMU havingcompletely removed the device from the PCI bus. This probably has todo PCI bus hotplug handlers (DeviceState->BusState->hotplug_handler).
So we invoke HasPCIDevice up to 5 times with 1 sec sleep in between...
Update hotplug design doc
..to reflect the transition from HMP to QMP during hotplugactions.
qa: Re-enable NIC hotplug tests
This was disabled due to Issue 885. This issue states that NIChotplugging sometimes makes an instance unresponsive. This cannot bethe case, since NIC hotplug takes place on an extra NIC and does notmess up with instance's primary NIC. Still this sometimes might be...
qmp: Log qmp commands and responses
Add debug logging for all qmp commands and responses, except forquery-commands, and qmp_capabilities, that are executed uponinitialization of QmpConnection (and their output is not reallyuseful).
hotplug: Is not supported for QEMU < 1.7
Change the bulk check of whether hotplug is supported or not.Only versions >= 1.7 support the required qmp commands.
hotplug: Remove unused code after refacoring
All HMP related code gets removed since everything is done via QMP.
hotplug: Use QMP during HotDelDevice
During device hot-del use new QMP helper methods: HotDelDisk() fordisk hotplug and HotDelNic() for NIC hotplug.
hotplug: Use QMP during HotAddDevice
During device hot-add use new QMP helper methods: GetFreePCISlot()for allocating a free PCI slot, HotAddDisk() for disk hotplug andHotAddNic() for NIC hotplug.
kvm: Use the new interface during instance start
Use new OpenTap() and _GetNetworkDeviceFeatures() interfacesduring _ExecuteKVMRuntime().
monitor: Extend HotAddNic() for advanced features
In case vhost is enabled we have to pass the vhostfds as returned byOpenTap() via SCM_RIGHTS prior issuing netdev_add command with vhostand vhostfds extra arguments.
In case multiqueue is enabled we add the mq and vectors extra...
kvm: Refactor _GetNetworkDeviceFeatures() helper
Make _GetNetworkDeviceFeatures() return a tuple of (dict, str, str).The first item should be a dictionary including the enablednetdev/device features (vhost, vnet_hdr, mq). The other two arestrings to be appended to --netdev and --device qemu options....
netdev: Refactor OpenTap for future use
Make OpenTap() helper method to take a dictionary with the featuresthat should be enabled or not. Currently these features are: vhost,vnet_hdr, multiqueue.
In case vhost is enabled, along with the tapfds we return a list of...
hotplug: Use QMP in VerifyHotplugCommand
This used to check whether a specific hotplug action has succeeded.Use HasPCIDevice() qmp method to verify if the requested deviceexists or not.
hotplug: Use QMP in VerifyHotplugSupport
This used to check whether a specific hotplug action is supported.Since everything is about to be done via qmp commands delegate thesechecks to QmpConnection class.
monitor: Close socket fd if already connected
We close the socket fd only if we have an active connection.We hide this logic in the public close() method, and allthe actions are moved to a private method.
kvm: New _with_qmp decorator
The decorator takes the hypervisor and the instance object as passedto all hotplug related commands and ensures a valid QmpConnection()object is found in hypervisor's qmp attribute. After the methodreturns, the qmp connection is closed. This way if we have nested...
qmp: Add Disk hotplugging related methods
HotAddDisk() uses blockdev-add and device_add after getting thedrive fd and passing it with the AddFd() helper method. Please notethat in case of userspace access mode this step will be omitted.
HotDelDisk() uses device_del. Please note that drive_del is...
qmp: Add NIC hotplugging related methods
HotAddNic() uses netdev_add and device_add after passing the tapfdto the running process with the GetFd() helper method.
HotDelNic() uses device_del and netdev_del qmp commands.
qmp: Helper methods for parsing query-pci output
Introduce GetPCIDevices() that gets the devices of the first PCI busof a running instance, HasPCIDevice() that will be used to verifyif a specific device is correctly hotplugged or not, andGetFreePCISlot() that will be used during hot-add to find the first...
qmp: Add helper methods to verify hotplug support
Introduce CheckDiskHotAddSupport() and CheckNicHotAddSupport()helper methods that will be used to verify hotplug support.Both check for fdsend python module and if the required qmpcommands are supported....
qmp: Add GetFd() wrapper around getfd command
getfd qmp command will be used to pass an fd using SCM_RIGHTSand name it properly so that NIC hot-add can take place.
qmp: Refactor of add-fd and remove-fd commands
Instead of sending the qmp command along with the SCM_RIGHTS controlmessage we send just a blank (idea taken from qemu iotests). Then weinvoke the corresponding command (add-fd, remove-fd) with theregular way (qmp.Execute). Since the qmp connection does not close...
utils: Introduce GetFreeSlot() function
Since this is a generic function that implements bitarray logic moveit from kvm to utils so that it can be easily used across all modules.
Make it raise errors.GenericError if it cannot find a free slot inthe given bitarray....
kvm: Minor refactor of MonitorSocket
Create socket upon connect(), keep the status of connection(self._connected = False) in close(), make existing connect()method protected and let connect() act as a reconnect method.
Link to the config-lock design document
Merge branch 'stable-2.12' into master
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
The config lock still exists
...so update the daemons-refactoring design document accordinglyto fit with implementation. The mentioned design document alreadyexists on the master branch.
Add a design document for removal of the config lock
This design document describes how the negative effect of the configlock can be removed in an incremental way.
qa_filters: Use retry in more cases
Even starting a (delay 0.01) job without filters does not happenwithin the 0.5 seconds we allowed if run in a big cluster.Use retry here as well.
Signed-off-by: Niklas Hambuechen <niklash@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
qa: qa_filters: Make status check less flaky by using retry
The time needed to delete a filter and have jobs that were pausedby that filter be scheduled depends on the size of the cluster.
The polling retry introduced here is more robust than a sleep....
qa: qa_job_utils: add `RetryingUntilJobStatus`
Dual to `RetryingWhileJobStatus`.It retries until the given status is reached.
QA: Move `GetJobStatus`/`RetryingWhileJobStatus` to utils
We would like to use them from other QAs involving jobs.
JQScheduler: Run scheduler when filters change in config
Until now, job filters being added, modified, or deleted triggereda scheduler run (`scheduleSomeJobs`) from the place where the filtermodification request is handled (Query.Server).
This could lead to a race:...
JQScheduler: Allow triggering scheduler if config changes
Some changes to the config need the scheduler to be re-run,e.g. when job filters change.
This commit adds the infrastructure for that, with no thingstriggering rescheduling.
Signed-off-by: Niklas Hambuechen <niklash@google.com>...
ConfigReader: Simplify `initConfigReader` argument passing
We passed in a cfg_transform and an IORef, just to apply thecfg_transform and write it to the IORef.
Instead, directly pass in a function that decides what to dowith the read config, so that `ConfigReader` now is solely...
Fix API doc
...by also removing the unused parameter from the doc string.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Niklas Hambuechen <niklash@google.com>
Query.Server: Fix GHC 7.8 `bitSize` deprecation warning
This makes Ganeti compile on GHC 7.8 with our -Werror.
Signed-off-by: Niklas Hambuechen <niklash@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Drop unused argument from RunSshCmdWithStdin
Due to merging up 9f8f740 the ssconf_store (positional) argumentbecame unused in 6a06c42. So remove it from the interface.
Drop ipv6 keyword argument for SshRunner
...as the SshRunner now determines by itself the addressfamily. This is a semantic conflict of merging 9f8f740into master.
Conflicts: lib/bootstrap.py...
Merge branch 'stable-2.11' into stable-2.12
Conflicts: lib/client/gnt_instance.pyResolution: follow the client switch...
Make ConfD honor /etc/services
...to determine which port to bind to. The priorities are now- the --port option,- the services entry, and- the default port.
break too long line
Merge branch 'stable-2.10' into stable-2.11
Conflicts: lib/client/gnt_instance.pyResolution: follow the switch to query socket
QA for job filtering
Includes QA for RAPI filter management and gnt_filter.
System tests (adding/removing filters, observing their effects)are done with gnt_filter.
qa_utils: Allow passing fail=None to AssertCommand
This is for cases where we don't care about the exit code.
qa_utils: Make AssertCommand return stdout/stderr as well
This is usefull if the run commands output something that's needed afterwards.
Unit tests for job filtering
This tests only the filter logic; the QA testing filter management + logiccomes later.
Implement job filtering
This implements the operational part of the design doc "Filtering of jobs for the Ganeti job queue" (design-optables.rst).
It includes- respecting filter rules when jobs are scheduled- cancelling running jobs rejected by filters...
JSON: Add `nestedAccessByKey` and `nestedAccessByKeyDotted`
We will need them for job filters, where we want to access OpCodesas a JSON structure.
Query.Filter: Add `evaluateFilterJSON`
We will need it to evaluate filters on OpCodes for job filtering.
Objects: Add filterRuleOrder / Ord instance for FilterRule
This is necessary to process filters in this order, as requiredby the filtering spec.
Look up RPC port
...instead of hard-codedly using the default port.Ganeti's noded honors /etc/services, so also lookup the port there when trying to contact the daemon.
Add withDefaultOnIOError
This is a specialized version of try with a constanthandler. The main benefit is the type annotation forcingthe free type variable in 'try' to be IOError.
Read if IPv6 is used directly in SshRunner
Moved check if the Primary Network is running on IPv6 into the SshRunner.Until now SshRunner was only initialized correctly in the bootstrap.pyand not in the other 5 places.
Fixing issue #892
Signed-off-by: Neal Oakey <neal.oakey@googlemail.com>...
Test.JQScheduler: Fix queue jobs having same job IDs
The `Arbitrary Queue` instance so far would simply generated listsof jobs, for which it was possible that job IDs are repeated(inside individual lists or accross all lists).
While this hasn't created any problem so far, real-world Queues...
TestCommon: Add listOfUniqueBy
This is a Gen for creating elements that are distinct by a certain property,(e.g. for generating a list of jobs, all of which have different job IDs).
Utils.Monad: Add unfoldrM and unfoldrM'
Taken from monad-loops, with a variable rename to avoid name shadowing.
Fix FIXME comment
As Ganeti supports jump-upgrades, we cannot anywhere in the2.x series assume that we have not just been upgraded forma 2.10 or earlier system. Hence the bootstrap code needs toremain. Change the comment accordingly.
Retry death detection after killing
We cannot avoid the race on death detection afterforcefully killing a job: the only guarantee theoperating system gives us is that the process willdie eventually. However, we can improve the chanceof being able to successfully clean up a job by...
Add orM from monad-loops
...a monadic short-circuit version of 'or'.
Add constants for retries of death detection
Add constants describing how often, and at which intervals,to retry death detection of a job after sending it a SIGKILL.
Report back death status after check
Make cleanupIfDead report the death status of the job,so that a caller can decide to retry.
Improve debug statements in death detection
Also clean up stale livelock files
So far, WConfD only cleans up the livelock files of resourceowners. While this is sufficient for reclaiming resources andGaneti to work properly, we still don't want to leak stalelivelock files.
Add a function to list livelock files found on the system
While we mainly care to detect deaths of resource owners, wealso want to avoid leaking stale livelock files. Those canoccur is someone dies after returning all resources. Providea function listing them, so that they can be cleaned up....
Check the return value of query in ConnectToInstanceConsole
When running gnt-instance console for an instance name that doesn'texist, I got the following raw error:
... cl.QueryInstances([instance_name], ["console", "oper_state"], False)ValueError: need more than 0 values to unpack...
Test.JQScheduler: Make test more precise
The output of a filter operation should be exactly a subsequence.
Utils: Add `isSubsequenceOf`
It is useful for defining properties of `filter` like functions(in fact, it is the only property that can be said about filterif the filter predicate is not known).
Test.JQScheduler: Refactor enqueued jobs
Makes the test easier to read.
« Previous 1 2 3 4 5 ... 161 Next » (201-300/16089) | Per page: 25, 50, 100
Also available in: Atom