Ganeti 2.0 cluster parameters design doc
Reviewed-by: ultrotter
Implement job 'waiting' status
Background: when we have multiple jobs in the queue (more than just afew), many of the jobs (up to the number of threads) will be in state'running', although many of them could be actually blocked, waiting forsome locks. This is not good, as one cannot easily see what is...
OS Interface design doc
Reviewed-by: imsnah
Add .. contents:: marker to design docs
Implement job auto-archiving
This patch adds a new luxi call that implements auto-archiving of jobsolder than a certain age (or -1 for all completed jobs), and the gnt-jobcommand that makes use of this (with 'all' for -1).
Add a simple timespec parsing function
This function will be used for auto-archiving jobs via the command line.The function is pretty simple, we only support up to weeks since monthsand higher are not 'precise' entities, and dealing with them wouldrequire us to start using calendar functions....
backend.py change to get cluster name from master
Currently there are three function in backend that need the cluster namein order to instantiate an SshRunner. The patch changes these to get thecluster name from the master in the rpc call; once the multi-hypervisor...
Disable re-reading of config file
Since the objects read from the config file are passed to the variousthreads, it's unsafe to re-read the config file (and throw awayConfigWriter._config_data). As such, we disable the re-reading of thefile (since now the master is the owner the file, it makes not sense to...
RAPI Desing Doc
Reviewed-by: iustinp
Start implementation of parallel burnin
This patch introduces a simple framework for executing jobs in parallelin burnin (the ExecJobSet function) and the "--parallel" command lineflag.
The patch also changes the instance creation to run in parallel when the...
Fix gnt-job list with empty timestamps
In case the job object doesn't have a timestamp (which is a separateissue), the listing should not break. We fix this by changing theFormatTimstamp function itself to return '?' in case the timestampdoesn't look good (note that it still can break if non-integers are...
Increase the number of threads to 25
Since our locks are not gathered nicely, we can have jobs that areactually blocking on locks (parallel burnin shows this), so at least weneed to increase the number of threads above the usual number of jobs wecould have in a such a case....
Minor cleanups & typo fixes.
Fix SshRunner breakage from the changed API
More places actually use the SshRunner than just the gnt-clustercommands.
Change SshRunner usage
Currently the SshRunner uses a SimpleConfigReader instance, however thisis not best. We change it to use the cluster name directly (and itsconstructor now takes this as parameter, instead of SCR), and itscallers are change to pass the name directly....
Update document describing cluster security
It may need further updates, but here's a start.
Fix ssconf.GetMasterAndMyself
The ssconf migration left this out.
Reviwed-by: imsnah,ultrotter
Fix a mistake in the gnt-backup man page
The actual location is /export, not /exports.
Use docbook2* paths found during configure for actual build
docbook-wrapper had the names for the docbook2* programs hardcoded. Thispatch changes Makefile.am and the wrapper script to pass them viaanother argument.
Another issue where rapi.in was built before rapi-resources.sgml is...
Remove references to Twisted framework
Get rid of ssconf
Remove leftovers from ssconf.
Don't pass sstore to LUs anymore
sstore is no longer used in LUs.
Convert ganeti-master
Use simpleconfig instead of ssconf.
Convert ganeti-watcher
Use RPC calls instead of ssconf.
Convert ganeti-noded
Replace ssconf with utility functions.
Convert gnt-cluster
Replace ssconf with configuration.
Convert bootstrap.py
Convert cmdlib.py
Replacing ssconf with configuration. Cluster rename is broken and staysthat way.
Convert ssh.py
Get rid of ssconf and convert to configuration instead.
Convert rpc.py
Replacing ssconf with utility functions.
Convert hypervisor
Replacing ssconf with configuration.
Convert mcpu.py
Convert config.py
The configuration version is now again in the configuration file.
Convert backend.py
Replacing ssconf with simpleconfig.
Add new query to get cluster config values
This can be used to retrieve certain cluster config values fromwithin clients.
OpDumpClusterConfig was not used anywhere, hence I'm just reusingit. The way ConfigWriter.DumpConfig returned the configurationwas not thread-safe, anyway (no deepcopy)....
Move functions from ssconf.py elsewhere
These functions will be used to access config values instead of usingssconf.
Add simple configuration reader/writer classes
This will be used to read the configuration file in the node daemon.The write functionality is needed for master failover.
Fix the watcher with down nodes
The watcher didn't handle the down nodes, fix this by ignoring (insecondary node reboot checks) any node that doesn't return a boot id.
Fix the watcher not restarting instance bug
The watcher was using conflicting attributes of the instance: - it queried the admin_/oper_state, which are booleans - but it compared those to the status (which is a text field)
The code was changed to query the aggregated 'status' field, as that...
Remove last use of utils.RunCmd from the watcher
The watcher has one last use of ganeti commands as opposed to sendingrequests via luxi. The patch changes this to use the cli functions.
The patch also has two other changes: - fix the docstring for OpVerifyDisks (found out while converting...
Fix unittests broken by revision 1727
Add cluster options from ssconf to configuration
ssconf will become write-only from ganeti-masterd's point of view,therefore all settings in there need to go into the main configurationfile.
Move instantiation of config into bootstrap.py
Future patches will add even more variables to the cluster config.Adding more parameters wouldn't make the function easier to use andit doesn't make sense to pass them to another function, as it'sonly done once in bootstrap.py on cluster initialization....
Change the results from cli.PollJob
Curently PollJob accepts a generic job, but will return (historyartifact) only the first opcode result. This is wrong, as it doesn'tallow polling of a job with multiple results.
Its only caller (for now) is also changed, so no functional changes...
Add list of build dependencies
Build HTML from RST input
This patch also adds the design documents to Makefile.am.
Fix ‘gnt-job info’ with no arguments
I didn't realize that my zip will break when no args are passed...
Add output of job/opcode timestamps
This patch adds posibility of selection of job/opcode timestamps ingnt-job list and info.
The code handling the possible cases (None or a valid timestamps) areugly though...
Reviwed-by: imsnah
Enhance the job-related timestamps
This patch adds start, stop, and received timestamp for jobs (and allowsquerying of them), and allows querying of the opcode timestamps.
Small fixes for master daemon design document
It said CLI/RAPI will talk to master using HTTP, which isn't true. Adda reference to job queue design document. Small typos.
Import design doc for commandline arguments
locking design: code path and declarations
locking design: explain use of async mode
Before we were discussing this possible future feature, and itsdrawbacks, but not its usefulness. This patch corrects this.
locking design: talk about removing locks
Import (and update) granular locking design doc
Abstract the timestamp formatting into cli.py
Currently we format the timestamp inside the gnt-job info function. Wewill need this more times in the future, so move it to cli.py as aseparate, exported function.
Add job queue design document
Add an 'index' of design documents
This will be an overview document, enumerating the changes without goinginto details and pointing to the actual documents.
Add opcode execution log in job info
This patch adds the job execution log in “gnt-job info” and also allowsits selection in “gnt-job list” (however here it's not very useful asit's not easy to parse). It does this by adding a new field in the queryjob call, named ‘oplog’....
Move a hardcoded constant to constants.py
For now we only use the ‘C’ protocol so we can put it in constants.pyinstead of hardcoding it.
Enable the use of shared secrets
This patch enables the use of the shared secrets for DRBD8 disks, using(hardcoded in constants.py) the md5 digest algorithm.
For making this more flexible, either we implement a cluster parameter(once the new model is in place), or we can make it ./configure-time...
Extend DRBD disks with shared secret attribute
This patch, which is similar to r1679 (Extend DRBD disks with minorsattribute), extends the logical and physical id of the DRBD disks with ashared secret attribute. This is generated at disk creation time and...
Add a info subcommand to gnt-job
Currently, it is hard to examine a job in detail; the output of ‘gnt-joblist’ is not easy to parse.
The patch adds a ‘gnt-job info’ command that is (vaguely) similar to‘gnt-instance info’ in that it shows in a somewhat easy to understand...
Implement job summary in gnt-job list
It is not currently possibly to show a summary of the job in the outputof “gnt-job list”. The closes is listing the whole opcode(s), but thatis too verbose. Also, the default output (id, status) is not veryuseful, unless one looks for (and knows about) an exact job ID....
Nicely sort the job list
Unless we decide to change the job identifiers to integer, we should atleast sort the list returned by _GetJobIDsUnlocked.
Move the pseudo-secret generation to utils.py
The bootstrap code needs a pseudo-secret and this is currently generatedinside the InitGanetiServerSetup function. Since more users will needthis, move it to utils.py
Fix a bug related to static minors
When the node does not yet have any minors allocated, the first minor(0) will not be entered in the ConfigWriter._temporary_drbds structure.This does not happen for our current usage, since we always ask for twominors (so the next call will not match this case), but it will be...
Add checks for tcp/udp port collisions
In case the config file is manually modified, or in case of bugs, thetcp/udp ports could be reused, which will create various problems(instances not able to start, or drbd disks not able to communicate).
This patch extends the ConfigWriter.VerifyConfig() method (which is used...
Update the cluster serial_no on certain operations
This patch adds update of the cluster serial number for: - add/remove node (as the cluster's node list is changed) - add/remove/rename instance (as the cluster's instance list is changed) - change the volume group name...
Allow listing of the serial_no via gnt-* list
This patch adds listing of the serial_no attribute in gnt-instance andgnt-node list, and updates to the manpages to reflect the change.
Initialize and update the serial_no on objects
This patch add initialization of the serial_no on instance and nodes,and update of the field whenever an object is updated in the genericcase, via ConfigWriter.Update(obj) and in the specific case ofinstances' state being modified manually....
Switch the global serial_no to the top object
Currently the serial_no that is incremented every time the configurationfile is written is located on the 'cluster' object in the configurationstructure. However, this is wrong as the cluster serial_no should be...
Add serial_no attributes to objects
This patch adds the ‘serial_no’ attribute to the other top-level objects(the configuration object itself, the nodes and the instances).
Replace a cfg.AddInstance with UpdateInstance
This seems to be the last (deprecated) use of AddInstance in order toupdate an instance.
The patch also removes a whitespace-at-eol case.
Add design doc for the disk changes
Fix iallocator name
port forward of patch from revision 1690 with following message:
Patch on revision 1686 used the wrong field: ial.name, which is the instancename and not the iallocator name. self.op.iallocator is the right field.
Sorry for this inconvenience....
Fix a broken format string
This patch fixes a broken format string. It's expecting 3 parameters, but onlygets 2. This change will add the missing parameter. This is a forward-portof the fix in Ganeti 1.2
Design doc for the master daemon
It's not complete, but I hope it's up to date. It's restructured text,but no make rules or such for html output are needed, it's readableas-is.
Switch config.py to logging
A couple of more modules are using the obsolete logger functions, configbeing one of them.
Switch to static minors for DRBD
With some todos remaining, this patch switches the DRBD devices to usethe passed minors, and the cmdlib code (add instance and replace disks)to request and assign minors to the DRBD disks.
Todos: - look at the disk RPC calls to see which can be optimized away, since...
Implement config support for drbd static minors
This patch adds support for allocating static minors.
Like for the LVM uuids, we add a new cache for the temporarily allocatedrequests, and the users of the new methods must manually clear thecache. If this doesn't happen, at worst we lose some minors....
Fix disk replace secondary with static minors
The code in 'updating instance configuration' section of the replacedisks with change secondary node was setting a wrong new logical_id forthe drbd devices (only set the new node, not the new minor). The patch...
Extend DRBD disks with minors attribute
This patch converts the DRBD disks to contain also a minor (per eachnode) attribute. This minor is not yet used and is always initializedwith None, so the patch does not have any real-world impact - except forautomatically upgrading config files (it adds the minors as None, None)....
Apply filter properly in LUQuery{Nodes, Instances}
Currently when not locking all nodes/instances are returned, regardlessif the user asked only for some of them. With this patch we return tothe previous behaviour: - if no names are specified return info on all current ones...
Remove auto_balance from burnin/cmdlib
There is no such feature in trunk yet.
Add utils.ReadFile function
It abstracts exception handling and is like a complement toutils.WriteFile.
doc/locking.txt: Job queue functions can be called from LUs
The previous description was unclear. Reported by Guido Trotter.
Make burnin aware of hvm device type flags.
Merged from branches/ganeti/ganeti-1.2 r1648Use static values for new hvm instance flags
GetAllInstancesInfo, change internal iterator name
GetAllInstancesInfo used "node" as an iterator name. Change it toinstance to make it less confusing.
gnt-instance: fix tags commands online help
It used to refer to "nodes", which was confusing.
Parallelize Tag operations
For now we lock the instance/node for adding/deleting tags from it, butwe could probably in the future do without, with more support from theconfig for atomic operations.
Parallelize LUSetClusterParams (and add a FIXME)
Fix a typo in gnt-instance.sgml
Parallelize LURemoveExport
Parallelize LURemoveInstance
Using the new add/remove infrastructure this becomes pretty easy! :)
Parallelize LUCreateInstance
Finally, instance create on different node, without iallocator, can runin parallel. Iallocator usage still needs all nodes to be locked,unfortunately. As a bonus most checks which could have been moved toExpandNames, before any locking is done....
Implement adding/removal of locks by declaration
With this patch LUs can declare locks to be added when they start and/orremoved after they finish. For now locks can only be added in theacquired state, and removed if owned, and added locks default to be...
LockSet: forbid add() on a partially owned set
This patch bans add() on a half-acquired set. This behavior waspreviously possible, but created a deadlock if someone tried to acquirethe set-lock in the meantime, and thus is now forbidden. ThetestAddRemove unit test is fixed for this new behavior, and includes a...
Fix typo in a locking.py comment
Use is_owned to determine whether to unlock
Now that is_owned is public we don't need to play games at the end of anLU. If we're still owning anything we just release it.