First version of user feedback fixes
This patch contains a raw version for fixing feedback_fn.
The new mechanism works as follows: - instead of a per-Processor feedback_fn, there's one for each ExecOpCode, so that feedback for different opcodes go via possibly...
Cache some jobs in memory
This patch adds a caching mechanisms to the JobStorage. Note that isdoes not make the memory cache authoritative.
The algorithm is: - all jobs loaded from disks are entered in the cache - all new jobs are entered in the cache...
Fix JobStorage._GetJobIDsUnlocked
The job ID returned must be an integer (and the regex enforces that),but we didn't convert it manually.
Reviewed-by: imsnah
Change JobStorage to work with ids not filenames
Currently some of the functions in JobStorage work with filenames (whichis an implementation detail and should only be used when dealing withthe storage) and not with job IDs. We need to change this in order to...
Add experimental persistency to job queue
It's not perfect and it's not finished, but it's a start.
- Serial number is read only once, but written on each update- Jobs are kept only on disk (caching will be implemented)
Reviewed-by: iustinp
Convert backend.py to the logging module
The patch also switches some of the exception logs to uselogging.exception (and therefore the log message will have a diferentformat).
(Note that this might not be a good choice in all cases, though)
Add PID to all logs
This patch (for trunk) adds the PID to all daemon logs.
Fix backend.NodeVolumes handling of LVM output
This is the same fix as for GetVolumeList.
I've checked manually and all other places that call lvm commands arealready checking the output validity in terms of correct number offields.
Reviewed-by: ultrotter
Fix backend.GetVolumeList handling of LVM output
Sometimes ‘lvs’ can spit error messages on stdout, even when one wantsto parse the output:...Inconsistent metadata copies found - updating to use version 2776...
So we need to validate the output to guard against such cases....
Add generic HTTP server classes
Some of the code is adopted from the 1.2 branch(lib/rapi/RESTHTTPServer.py). This code can be used as a base for thevarious HTTP servers in Ganeti.
Make "gnt-job list" work again
"gnt-job list" was broken after my recent changes in the RPCbetween clients and the master. This patch makes it work again.
Initial copy of RAPI filebase to the trunk
Move watcher's LockFile function to utils
Switch _QueuedOpCode to have their own lock
Right now, the queued opcode doesn't have a lock, and instead relies onthe parent QueuedJob's lock.
This is not good for logging feedback, so it's better to have a lock foreach queuedopcode.
Add a simple decorator for instance methods
This is just a simple, hardcoded decorator for object methods needingsynchronization on the _lock instance attribute.
jqueue: Log more information when running opcodes
Fix double-logging in daemons
Currently, in debug mode, both the logfile handler and the stderrhandler will log debug messages. Since the stderr is redirected to thesame logfile (to catch non-logged errors), it means log entries aredoubled.
The patch adds an extra parameter to the logger.SetupDaemon() function...
Move the master socket in the ganeti run dir
... as it was intended from the beggining, but by mistake left in thetop run dir.
Reduce duplicate Attach() calls in bdev
Currently, the 'public' functions of bdev (FindDevice andAttachOrAssemble) will call the Attach() method right after classinstantiation.
But the constructor itself calls this function, and therefore we haveduplicate Attach() calls (which are not cheap at all)....
Convert bdev.py to the logging module
This does not enhance in any way the messages; it just switches to thenew module.
Convert utils.py to the logging module
The patch also logs all commands executed from RunCmd when we are atdebug level.
Remove the old locking functions
This removes (hopefully) all traces of the old locking functions anduses.
Remove old job queue code
Change masterd/client RPC protocol
- Introduce abstraction class on client side- Use constants for method names- Adopt legacy function SubmitOpCode to use it
Make luxi RPC more flexible
- Use constants for dict entries- Handle exceptions on server side- Rename client function to CallMethod to match server side naming
Add very simple job queue
Convert LUTestDelay to concurrent usage
In order to do so: - We set REQ_BGL to False - We implement ExpandNames
That's it, really.
Processor: Acquire locks before executing an LU
If we're running in a "new style" LU we may need some locks, as requiredby the ExpandNames function, to be able to run. We'll walk up the locklevels present in the needed_locks dictionary and acquire them, then run...
LogicalUnit: add ExpandNames function
New concurrent LUs will need to call ExpandNames so that any namespassed in by the user are canonicalized, and can be used by hooks,locking and other parts of the code. This was done in CheckPrereqbefore, but it's now splitted out, as it's needed for locking, which in...
Processor: Move LU execution to its own method
This makes the try...finally code simplier, and helps adding a morecomplex locking structure before the actual execution. It also fixes aconcurrency bug caused by the fact that write_count was read beforeacquiring the BGL, and thus spurious config update hooks run could have...
constants: Add job and opcode status strings
workerpool: Don't notify if there was no task
Workers have to notify their pool if they finished a task to makethe WorkerPool.Quiesce function work. This is done in the finally:clause to notify even in case of an exception. However, beforewe notified on each run, even if there was no task, thereby creating...
Add a top level RUN_GANETI_DIR constant
This patch creates a base RUN_GANETI_DIR and then moves the other rundir constants to use that (even if just setting BDEV_CACHE_DIR as equalto it, rather than putting it deeper, for now).
Also we create a constant list of all the subdirs we need in RUN_DIR to...
symlinks: Add DISK_LINKS_DIR constant
The DISK_LINKS_DIR points to the RUN_DIR/ganeti/instance-disksdirectory, which will contain symlinks to the instances' disks. Theseprovide a stable name accross all nodes for them, and permitlive-migration to happen....
luxi: Use serializer module instead of simplejson
serializer.DumpJson: Control indentation by parameter
If the simplejson module supports indentation, it's always used. Thereare cases where we might not want to use it or enable it only fordebugging purposes, such as in RPC.
Add a missing import to cmdlib
cmdlib uses some constants from locking (ie. locking levels) but doesn'timport it. This patch fixes the issue.
Fix an error accessing the cfg
Since the context is passed to LogicalUnit, rather than the cfg, we canonly access the cfg as self.cfg, self.context.cfg, or context.cfg (inthe constructor). cfg is not valid anymore.
Add and remove instance/node locks
Whenever we add an instance or node to the cluster (i.e. to the configand whenever we remove them we should add/remove locks as well). In thefuture we may want to optimize this so that the configwriter does it, orit's handled at the context level, but till we're adding/removing...
Pass context to LUs
Rather than passing a ConfigWriter to the LUs we'll pass the wholecontext, from which a ConfigWriter can be extracted, but we can alsoaccess the GanetiLockManager. This also fixes the places where a FakeLUis created.
Fix a typo in LUTestDelay docstring
Locking: remove LEVEL_CONFIG lockset
Since the ConfigWriter now handles its own locking it's not necessary tohave a specific level for the config in the Locking Manager anymore.This patch thus removes it, and all the unittest calls that used it, ordepended on it being present....
ConfigWriter: synchronize access
Since we share the ConfigWriter we need somehow to make sure thataccessing it is properly synchronized. We'll do it using thelocking.ssynchronized decorator and a module-private shared lock.
This patch also renames a few functions, which were called inside the...
Locking: add ssynchronized decorator
This patch creates a new decorator function ssynchronized in the lockinglibrary, which takes as input a SharedLock, and synchronizes access tothe decorated functions using it. The usual SharedLock semantics apply,so it's possible to call more than one synchronized function at the same...
ConfigWriter: remove _ReleaseLock
Remove empty function _ReleaseLock and all its calls. Since we onlyhave one configwriter per cluster the locking needs to cover all thedata in the object, and not just the file contents. Locking inConfigWriter will be handled using the ganeti locking library....
Add generic worker pool implementation
Reuse the luxi client in cli.SubmitOpCode
By a mistake, we don't reuse the luxi client. As such, we open and closethe connection at each poll cycle and spam the server logs.
Add custom logging setup for daemons
It's better for daemons if: - they log only to one log file - the log level is included - for debug runs, the filename/line number is included
This patch moves the custom formatter from the watcher to the logging...
Context: s/GLM/glm/
Make the GanetiLockManager instance of GanetiContext lowercase
Add a FirstFree function to utils.py
This function will return the first unused integer based on a list ofused integers (e.g. [0, 1, 3] will return 2).
Processor: acquire the BGL for LUs requiring it
If a LU required the BGL (all LUs do, right now, by default) we'llacquire it in the Processor before starting them. For LUs that don'twe'll still acquire it, but in a shared fashion, so that they cannot run...
Processor: pass context in and use it.
The processor used to create a new ConfigWriter when it was initialized.We now have one in the context, so we'll just recycle it. First of allwe'll pass the context in when creating a new Processor object, thenwe'll just use context.cfg, which is granted to be initialized, wherever...
Add REQ_BGL LogicalUnit run requirement
When logical units have REQ_BGL set (it is currently the default) theyneed to be the only ganeti operation run on the cluster, and we'llguarantee it at the master daemon level. Currently only one thread isrunning at a time, so this requirement is never broken....
Fix sstore handling in Processor
- no need to keep the sstore as an object member, remove it- don't reinitialize sstore only if self.cfg is None This is not an issue, as the Processor is recycled for every opcode, but in general we know that (a) we might need a different type of...
AddNode: move the initial setup to boostrap
From the master node we can't start ssh and connect to the remote node,nor we can do it from ganeti-noded as this ssh section will possibly askfor key confirmation and password. So the code to copy the ganeti-noded...
LUAddNode: use node-verify to check node hostname
As we can't use ssh.VerifyNodeHostname directly, we'll set up a mininode-verify to do checking between the master and the new node. In thefuture networking checks, or more nodes, can be added as well.
LUAddNode: use self.sstore, not a local ss
Since we're inside a LU we have access to self.sstore.No need to use ss, which separate instantiation will disappear in a fewpatches! ;)
LUAddNode: upload files via rpc, not scp
We used to scp all the ssconf files, and the vnc password file to thenew node. With this patch we use the upload_file rpc, specifying justthe new node as a destination. All the files previously copied by scpare already allowed by the backend....
Allow VNC_PASSWORD_FILE to be rpc-uploaded
What could possibly go wrong?
Change fping to TcpPing in two LUs
Two LUs are using RunCmd to call fping, in order to check for an IPpresence on the network. Substituting it with TcpPing will get rid ofit, which makes it not break in the new world order, where the mastercannot fork....
raise QuitGanetiException in LeaveCluster
Simplify QuitGanetiException instantiation
Rather than packing all the arguments in a tuple, let's pass themplainly. The superclass won't complain.
logger: Set formatter for stderr
Having a timestamp on log messages is very useful. The defaultformat string doesn't include a timestamp.
When removing a node don't ssh to it
Even in 1.2 this behaviour is broken, as the rpc call will remove thessh keys before we get a chance to log in. Now the rpc takes care ofshutting down the node daemon as well, so we definitely can avoid this.
This makes the LURemoveNode operation work again with the threaded...
Add errors.QuitGanetiException
This exception does not signal an error but serves the purpose of makingthe ganeti daemon shut down after handling a request. Currently it willbe used by ganeti-noded but in the future ganeti-masterd might make useof it as well. Its usage is documented in the docstring....
Add missing empty line in SshKeyError's docstring
Remove spurious check during LUAddNode
There is no point in checking whether the cluster VNC password fileexists as a prerequisite for AddNode, considering the check happens onthe master node, not the target one. Removing this check.
Improve LURemoveNode BuildHooksEnv docstring
Cleanup old DRBD 0.7.x code
Apparently there were still some leftovers. While removing an instance,I got the message "unhandled exception 'module' object has no attribute'LD_MD_R1'".
Cleanup LV status computation
Currently, when seeing if a LV is degraded or not (i.e. virtual volume),we first attach to the device (which does an lvdisplay), then do a lvsin order to display the lv_attr. This generates two external commands todo (almost) the same thing....
Remove lib/Makefile.libcommon
Fix gnt-cluster “command” and “copyfile”
Since the disabling of forking in the master daemon, the two ssh-basedsubcommands were not working anymore. However, there is no need at allfor the commands to be run from the master daemon (permissions to readthe cluster private ssh key notwithstanding), they can be run directly...
objects: Remove config_version from cluster configuration
Add functions to calculate version number to constants.py
In cfgupgrade, we need to extract parts of and build new version numbers.
utils.WriteFile: Remove optional check_abspath parameter
cfgupgrade will not work with relative paths at all, but rather get themfrom constants.py.
Add a ‘tags’ field to instance and node listing
Currently there isn't any easy way to list all nodes or instance andtheir tags; you have to query each node in turn, or list all the tagsvia something like “gnt-cluster search-tags '.*'”. Of course, this is...
Implement handling of luxi errors in cli.py
Currently the generic handling of ganeti errors in cli.py (GenericMainand FormatError) only handles the core ganeti errors, and not the clientprotocol errors (which live in a separate hierarchy).
This patch adds handling of luxi errors too, and also adds another luxi...
Add a rpc call for BlockDev.Close()
This patch adds rpc layer calls (in rpc.py and the equivalent inganeti-noded) to close a list of block devices, and the wrapper inbackend.py that takes a list of Disk objects, identifies them andreturns correctly formatted results....
Use a single Makefile.am instead of many
This change allows us to use cleaner dependencies betweendirectories. The build system is basically rewritten in large partsand may contain bugs.
Rework the DRBD8 device status computation
Currently, compute the status of a drbd8 device in GetSyncStatus andreturn only the values that we need (and fit in the framework ofGetSyncStatus). However, the full status details are useful (and needed)in other places, so the patch attempts to improve this situation....
ganeti-watcher: Replace custom exceptions with ganeti.error.*
Add more parameters to utils.WriteFile
- Make closing file optional: Required by ganeti-watcher to keep file open after writing it. Changes return value of utils.WriteFile if "close" parameter evaluates to True.- Pre- and post-write functions: Can be used to lock files. This...
Replace custom logging code in watcher with logging module
- Log timestamp for all messages- Write everything to logfile and optionally to stderr- Log messages are no longer buffered, allowing a user to see progress
Make sure serialized data ends with EOL character
Also fix the regular expression to not remove newlines. The simplejsonmodule puts whitespace at line endings when using indentation. Removeunnecessary import of ConfigParser module.
Allow disk object to set their own physical ID
Currently, the way to customize a DRBD disk from (node name 1, node name2, port) to (ip1, port, ip2, port) is to use the ConfigWriter methodSetDiskID. However, since this needs a ConfigWriter object, it can be...
Fix an error-handling case
There is a mistake in handling grow-disk for an invalid disk. This patchfixes it.
Implement disk grow at LU level
This patch adds a new opcode and LU for growing an instance's disk.
The opcode allows growing only one disk at time, and will throw an errorif the operation fails midway (e.g. on the primary node after it hasbeen increased on the secondary node). As such, it might actually leave...
Add method to update a disk object size
This patch adds a method that implements updating of a disk(object.Disk) size, together with its children.
While this will not track the exact disk size, it allows at least anapproximate size to be recorded in the configuration (and queried)....
Implement block device grow at the rpc layer
This simple patch exposes the block device grow operation at the rpclayer. It does not increase the protocol version as it has been recentlychanged by the live failover rpc call.
Expose block device grow in backend.py
This patch adds a wrapper over the block device grow operation thatconverts the input and output parameters as needed for the rpc layer.
bdev: implement disk resize for lvm/drbd8
This patch implements disk resize at the bdev level for the LVM andDRBD8 disk types. It is not implemented for DRBD7 and MD since the wayMD works with its underlaying devices makes it harder and thiscombination is also deprecated....
Move SetKey to WritableSimpleStore and use it
Before we used to be able to update SimpleStore by just calling SetKey, thisfeature is now moved to an external class, which inherits from it. In thispatch the new WritableSimpleStore class is also put to use, in the LUs that...
Add migration support at the rpc layer
This patch adds the migration rpc call and its implementation in thebackend. The patch does not deal with the correct activation of disks.
Because of the new RPC, the protocol version is increased.
hypervisor: add live migration support
This is just the hypervisor-level migration (e.g. “xm migrate”) not thewhole node coordination work.
Activate down instances' disks on replace-disks
When replacing disks or evacuating nodes with instances administrativelydown ganeti fails because the instance disks are not active. This patchactivates them, performs the replacement, and shuts them down again....
FailoverInstance: change AddInstance with Update
We're not adding a new instance, just making configuration changes tothe one we're working on.
Fix an error message in instance add
There is a mistake in the error message generated when we can't reach anode for checking for available disk space. Without it, the errormessage is:Failure: prerequisites not met for this operation:Cannot get current information from node '{u'gnte2.lab.k1024.org':...
Replace logging functions with calls to logging module
- Shorter code- Reorder arguments to logger.SetupLogging calls to make more sense
Fix a typo in jqueue.py
s/result/op_result/ (this code was never used, so this wasn't caught)
Move InitCluster opcode into a single function
This allows us to initialize a new cluster. The code certainly containsbugs and hooks aren't implemented yet.