QMP: make QmpConnection act as a context manager
This will allow automatic connection and socket cleanup on commandcompletion.
We also repeat the Qmp tests using the context manager. For this to befeasible, we move the test scenario to class variables and modify...
Complete zeroing
This patch will do all that is necessary to allow the helper VM to dowhatever it will do with the zeroing image in place - which is,hopefully, to zero out the disks of the instance.
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Add timeout parameters to gnt-backup export
This patch adds the two parameters to gnt-backup export, documentingtheir meanings in the manual file as well.
Add zeroing timeout params to opcode
This patch adds two parameters controlling the zeroing timeout - onethat is fixed and another that depends on the amount of data (size ofdisks) to zero.
Add zeroing function shell
This patch moves towards the actual zeroing by providing a functionthat determines the necessary size and creates and destroys a temporarydisk.
Add image size estimation function
To create a temporary disk, it is necessary to know just how big thisdisk must be. This patch adds a function that retrieves thisinformation for both data sources - a URL and a file.
Signed-off-by: Hrvoje Ribicic <riba@google.com>...
Add file info RPC call
Due to the need to determine the size of the temporary disk used forthe VM helper, an RPC call capable of supplying this information hasbeen added. It can be extended as needed to supply additionalinformation provided by Python's os.stat....
Modify failing assert
The assert stating that the disk did not have to be activated madesense once the instance's disks were never completely deactivated. Withzeroing, the assumption no longer holds true, and should be replaced bya check that the disks of the instance are indeed active....
Introduce class for temporary disk creation
The instance zeroing requires that a virtual appliance capable ofperforming it is used, and short of provisioning another VM, the bestGaneti can do is reuse the virtualization environment of the instance.To do this, a disk is needed to host the OS performing the zeroing, and...
Factor out and alter the instance running check
Examining whether an instance is running is useful outside the instancestate check, especially as the check relies on the admin_state recordedwithin the configuration, which the code treats as the desired rather...
Lock node if zeroing is used in LUBackupExport
This patch adds a node lock if zeroing is used, preventing anyoperations that might interfere with the amount of available space.
Add --zero-free-space option to gnt-backup export
This patch adds the option, allowing the CLI client to use the zeroingoption to save space when compressing images.
Add manual entry for gnt-cluster zeroing-image parameter
This patch adds a short description of the zeroing-image parameter tothe manual page of gnt-cluster.
Add the zeroing-image option
This patch adds the zeroing-image option to gnt-cluster and theOpBackupExport params. The many changes are all minor, yet necessary.
Factor out image validity check
The image validity check has been implemented as a part of the OSinstalls, yet it could be useful for the zeroing image as well. Thispatch factors the utility out.
Add HV parameter checks
The first version of zeroing requires two things: that theself-initiated shutdown of the instance can be detected, and that theinstance boots from the first disk supplied to it. This patch checksfor these by examining the hypervisor parameters....
Add the zero-free-space option
Although unused for now, add the zero-free-space option to the backupexport opcode, and add a rudimentary check for whether it should beused.
QMP: always return the command result
According to the QEMU Machine Protocol Specification, the messages sentby QMP as a response to a command can be of two types: either an errormessage (identified by the "error" key), or a success message(identified by the "return" key)....
QMP: keep greeting message version info
QMP will always report the QEMU version and package-specific string inthe greeting message, as per the QEMU Machine Protocol Specification. Westore this information and make it available to users of the monitor....
QMP: test supported_commands
Add test for supported_commands.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
QMP: raise exception on unsupported QMP command
Raise a special exception, qmp.QmpCommandNotSupported, whenever anunsupported QMP command is about to be executed. This is intended toassist in falling back to the human monitor for hotplug operations. As...
QMP: fetch supported commands on connect
Store the set of supported QMP commands inQmpConnection.supported_commands. This allows selective error handlingto give the caller a chance to downgrade to HMP when unsupportedcommands are encountered.
The `query-commands` QMP command used to implement this, appeared in...
Serve OS scripts from metadata daemon
This patch makes the metadata daemon web server serve the OS scriptsvia the instance communication mechanism. This is achieved byretrieving the "os" field from the instance parameters and serving thefile directly from the 'Constants.osSearchPath', typically...
Move processing OS params with visibility to web server
This will allow the metadata daemon to keep a more general instanceconfiguration and thus serve more general requests, such as, servinginformation related to the OS scripts.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>...
Move metad configuration related code to a separate module
... as an intermediate step before moving the responsbility ofcomputing the final OS parameter configuration from the configurationserver to the web server. This will allow the metadata daemon to...
Fix docstring type in 'FindFile'
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
re-add try statement
...that was lost in the merge
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Merge branch 'stable-2.11' into master
KVM: move tap control functions to a submodule
Move all tap-related functionality to the hv_kvm.netdev submodule. Werename _OpenTap to OpenTap, since it will now be used as a publicfunction.
Also, change the hv_kvm tests to import the new code.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>...
Convert QmpMessage to a new-style class
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
KVM: move monitor & QMP classes to a submodule
We move the HMP and QMP classes to the hv_kvm.monitor module.
Convert hv_kvm to a package
Weighing in at 3k LOC, it is probably time to split hv_kvm into a numberof modules before adding new code.
Merge branch 'stable-2.10' into stable-2.11
Switch to ClientType as identifier
...instead of Either String JobId.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Verify readJSON . showJSON = Ok for ClientType
Verify that the JSON encoding of the ClientType can correctlybe decoded.
Instance JSON ClientType
Make ClientType a JSON instance, in the way lock owners arealready encoded in JSON on the Python side.
Add a type for the entity owning a lock
While it is unlikely that lock owners will ever be represented bysomething different than a job id or, exceptionally, a string, thistype will still improves type checking. Additionally, it will allowus to make it a separate instance of JSON and thus simplify the...
Make LuxiD query WConfD for locks
Since WConfD is now the authoritative source for locks, make LuxiDquery this daemon for lock information rather than the master daemon.
Make locks field use live data
So far, the description of the locks fields was made under the assumptionthat lock queries wouldn't be answered by Luxid anyway, and hence it wasenough to parse such requests. However, now luxid will answer these queriesafter getting a snapshot of the locks status from wconfd. Hence make the fields...
Export lock name
...as locks are queried by name.
Add gnt-debug wconfd listalllocksowners
...so that wconfd's point of view on locks can be debuggeddirectly.
Expose listAllLocksOwners in WConfD
...so that LuxiD can ask for the lock status to answerlock queries.
Verify soundness of listAllLocksOwners
Verify that every owner mentioned in the output of listAllLocksOwnersactually owns the locks claimed.
Verify that all locks are mentioned in listAllLocksOwners
Add a test verifying that whenever an owner owns a lock, itis also mentioned in the output of listAllLocksOwners.
Verify consistency between listAllLocks{,Owners}
As listAllLocks is supposed to list all locks currently in use, this shouldbe a superset listAllLocksOwners, i.e., of all locks actively owned by someuser.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Add a function to provide all the full lock state
This function will be used to provide the information requiredto answer lock queries.
Add gnt-debug wconfd listalllocks
...to simplify debugging the lock status.
Expose listAllLocks on WConfD
Make wconfd provide information about the list of all locks currentlyin use.
Verify that the list of all locks is complete
Verify that all locks owned by someone are included in thelist of all (active) locks.
Add a function to obtain all active locks
Add a function that extracts from a LockAllocation the listof all locks that are in use (directly or indirectly).
Fix lint errors after moving Luxi code to UDSServer
Remove unused imports and update the 'buildCall' call in tests.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add a WConfd RPC Client
The client is generated automatically by Ganeti.THH.HsRPC.
It also includes helper functions for creating a client object.
Add a module for creating Haskell-to-Haskell RPC clients
It uses TH to create client functions with the corresponding signaturefrom a given list of server functions.
Add a generic TH currying function for arbitrary arity
.. and also a function for extracting a type argument from a type.
Move buildCall and validateResult from Luxi to UDSServer
.. to that they're available for other Luxi-like RPC clients as well.
While at it, rename validateResult to parseResponse to be consistentwith the names of the other functions.
Signed-off-by: Petr Pudlak <pudlak@google.com>...
Serve OS package from metadata daemon
... and fix an error message.
Fix metad port from 8080 to 80
Update 'daemons/daemon-util' to stop 'ganeti-metad'
Note that 'ganeti-metad' is not automatically started together withall the other daemons. Instead, 'ganeti-metad' is only started when...
Update metadata on create, import, reinstall, modify
Note that create and import are the same LU, therefore, there are atotal of 3 calls, not 4.
Add helper function to modify metadata
This function is responsible for transforming the instance into a dictand optionally overriding the OS parameters (public, private, secret).Also, in case of failure it issues a warning.
Add RPC to modify metadata
This RPC is used to send instance configuration data from the masterto the node daemon, which in turn sends that data to the metadatadaemon.
Use RPC transport to pass OS params to the metadata daemon
Add metadata daemon server entry point
... which starts the configuration and web servers.
Modify metadata daemon web server to serve OS parameters
Extend metadata web server to serve the OS parameters from the sharedMVar containing the instances' configurations, indexed by their IPaddress.
Add metadata daemon configuration server
The configuration server listens on a Unix socket for connections fromthe node daemon. The node daemon sends the instance parameters to theconfiguration server so they can be served through the metadata daemonweb server to the instances which have the communication mechanism...
Add metadata daemon types
The metadata daemon will be responsible for receiving the instanceparameters from the node daemon and keeping them in memory. The typeof the instance parameters in defined in a separate module because itwill be shared between the web server and the configuration server....
Move metadata daemon to its own directory
Currently, the metadata daemon is mainly a web server. However, a newconfiguration server will be introduced along with other metadatarelated modules. Therefore, this patch moves the current metadatadaemon web server to its own directory. This module is also renamed...
Add job execution to the daemons refactoring design doc
The section describes how Luxi daemon should execute the jobs so that itcan ensure it can detect their death at every point.
Fix the indentation of _LockList
.. to 2 spaces instead of 4.
Ensure that mcpu always works with lists of locks
Apparently sometimes unicode strings appear out of nowhere (maybe fromthe RPC calls) and then they are used as a list of characters (hailPython). This patch adds a function that ensures we always have a...
KVM: use running HVPs to calc blockdev options
Commit 0fe22ad2 moved the call to _GenerateKVMBlockDevicesOptions() from_GenerateKVMRuntime() to _ExecuteKvmRuntime. However, while in_GenerateKVMRuntime() there is only one set of HVPs, those of theinstance's configuration, in _ExecuteKVMRuntime there are two sets of...
KVM: reserve a PCI slot for the SCSI controller
Currently instances with disk_type=scsi are broken, because the SCSIcontroller uses a PCI slot not accounted for in the PCI assignmentlogic. We reserve a throw-away slot just for that.
This is a temporary workaround and will be reverted during the upcoming...
Check for LVM-based verification results only when enabled
This patch fixes a little glitch in 'gnt-cluster verify'.If LVM-based storage was disabled, it would still checkLVM-related verification results and print a confusingerror message.
Signed-off-by: Helga Velroyen <helgav@google.com>...
Fix "existing" typos
This patch fixes the wording of a couple of messages,including two typos of the word 'existing'.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Fix output of gnt-instance info after migration
After migrating a DRBD based instance, the output of gnt-instance infowas wrong wrt. DRBD minors. This patch fixes the output in such cases.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Upgrade Roman numerals support
Ganeti prouds itself of its really good retro-compatibility and API stability.Some of our users haven't upgraded their hardware in the last 2000 years (onecentury more, one century less) and their Xen-PVM (Parchment and VelluM) does...
Verify configuration version number before parsing
As the attempt to convert the dict used as json representationof the configuration into a configuration object already makesassumptions about the internal representation, verify the versionbefore such an attempt. Fixes issue 783....
Acquire config lock for RPC calls that check consistency
Since LUClusterVerifyGroup issues 3 RPC calls to get information fromnodes, and during every of these calls, the configuration can bemodified by somebody else, it's necessary to acquire a shared lock on...
Log asynchronous WConfd tasks and include the serial number
This makes debugging more informative and makes more clear howconfiguration distribution gets aggregated.
Make ConfigWriter internal lock re-entrant
The ConfigWriter now allows nested locking calls, and acquires(releases) the config lock from WConfd only the first (last) time.
This is needed for blocks that acquire a lock on the configuration anduse synchronized configuration calls inside. While this should be...
Enable logging of thread ID in Haskell daemons
Ignore private OS params in the get/put test
The get/put RAPI test tests for symmetry of what the RAPI output andwhat it takes in, with exceptions for legacy differences and optionsthat should not be exposed. The private OS params belong to thiscategory, and this patch adds them as an exception....
Improve the heuristic of guessing masterd's livelock file
As, during startup, masterd uses livelock files for specialtasks, the lexicographically last livelock file for masterdis the masterd_check_agreement_ file. However, when debugging,we want the primary livelock file. Hence extend the prefix to...
Make gnt-debug wconfd listlocks use only job ids (again)
Now that the product has been replaced by the sum we can, again,identify jobs only by their job id.
Use thread name as identifier for non-job tasks
Non-jobs requesting resources need to identify themselves by a stringdescribing uniquely what they do. For the time being (while masterdstill exists), these non-jobs are client requests to masterd. Fortunately,...
Identify lock owners by either string or job id
Most requests asking for resources are jobs. However, in exceptionalcases, other requests (like currently requests to masterd requiring theconfiguration) need to ask for resources. They identify themselves by...
Merge branch 'stable-2.9' into stable-2.10
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>...
Merge branch 'stable-2.8' into stable-2.9
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Fix failing gnt-node list-drbd command
The UUID/name switch avoided this particular bit of code, and as aresult the list-drbd command failed as it tried to compare UUIDs andnames. This patch fixes the retrieval, converts the newly returnedUUIDs to names, and modifies the QA to the results and not only the...
Remove VerifyConfig tests
These tests don't work with WConfd, because the configuration isverified automatically before each operation and it's not possible tomodify the configuration state directly.
It'll be necessary to re-create these tests for WConfd once...
Fix configuration calls for LUBackupExport
Re-read the instance in-memory objects from WConfd after calls thatchange its state.
Fix configuration calls for LUInstanceRename
Re-read the instance in-memory objects from WConfd after callsthat change its state.
Fix configuration calls for LUInstanceReboot
Fix configuration calls for LUInstanceReinstall
Fix configuration calls for LUInstanceShutdown
Fix configuration calls for LUInstanceStartup
Instance storage: Mark functions that modify instances
.. in the configuration with comments.
Also re-read an instance object where needed.