Use hypervisor / storage information only when requested
So far, the node queries ignored the list of fields andjust requested all available information from the backend.That means, for example if only hypervisor information isrequested, still the storage space calculation is...
Remove duplicate storage units in node query
This is a little performance tweak for the node queries.So far, the query code mapped the disk templates to storageunits. It could happen that two disk templates were mappedto the same storage unit and therefore the storage space...
build_chroot: for wheezy install base64-bytestring via cabal
The version packaged in debian wheezy is less than the minimallyrequired version 1.0.0.0.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Document the minimal version of base64-bytestring required
There exist several versions of base64-bytestring; the interface we'reusing is that of version 1.0.0.0 or higher.
Use the generic query field test for network QA
Instead of just using sporadic 'gnt-network list' commands,use the more exhaustive generic query test functions forlist and list-fields.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Update design doc wrt to queries from non-masters
This patch updates the design document of the querysplitting with respect to the proposal of making itpossible to run queries from other nodes than themaster node.
Signed-off-by: Helga Velroyen <helgav@google.com>...
Mention removal of python queries in NEWS
This patch mentions the removal of the python queries inthe NEWS file, including the removal of the--enable-split-queries option at configure time.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Make luxid job submission be defined by replication
When receiving jobs to be submitted, make luxid replicate them to allmaster candidates and then return. The actual execution can be handledasynchronously.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Add function to enqueue jobs
Add a function that ensures that a given set of jobs gets executed atthe appropriate time. At the moment, this is still the simplemechanism of handing over everything to masterd; but even at thisstage, it has the benefit of allowing to remove code duplication in...
Add a function justBad to filter the Bad value of a list
In the same way as justOk allows to filter the Ok values,add justBad to filter the Bad values. While there, simplifythe definition of justOk.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Add wrapper to replicate many jobs
Add a convenience wrapper around replicateJob to replicatemany jobs to the master candidates.
Add function to replicate a job to the master candidates
As luxid will be handling the job queue soon, add a utility toreplicate jobs to all master candidates. Also log errors.
Compress JobqueueUpdate RPCs
Noded understands compressed RPCs for updating files in the(replicated) job queue. Make use of this feature.
Add zlib and base64-bytestring as haskell dependencies
With luxid taking over more and more job management, replicating jobfiles should use the compressed option, as is currently done by masterd.This requires gzip compression and base64 encoding available in haskell...
Release internal lock for serial file later
When allocating new jobs, the new serial is replicated amongall master candidates. To avoid races with a later job idallocation, keep the internal lock till after the replicationattempt.
Rename LuxiSocket to MasterSocket
Rename the constants to name the socket to connect masterd,as the name LuxiSocket hints on luxid, which is differentfrom masterd.
Add gnt-network list to network QA
The QA for networks so far did not cover network queriesand as such issues with those could not be discovered.
build_chroot: Add packages for JSON patching (used in QA)
These (optional) packages allow patching QA JSON configuration file (see[e5398c3a]). While it's meant to be used on Buildbot, it can be usefulto have it installed locally too, for example for testing patches in a...
Instance queries: remove opcodes and LU
Removes the remains of the instance queries.
Export and network queries: remove opcodes and LUs
Removes the remains of the export (aka backup) and networkqueries.
Group queries: remove opcodes and LUs
Removes the remains of the group query code.
Node queries: remove opcodes and LUs
Removes the remains of the node query code.
Remove opcodes from haskell tests
Clears the haskell tests from the query opcodes that are nolonger needed.
Make burnin use luxi queries
Burnin was still using some op queries. With this patch,they are now replaced by luxi queries.
Remove op queries from masterd
Masterd still had some 'if' branches for the queries thatused to be op queries, but are now luxi queries. Thispatch removes them.
Remove instance query python code
This patch removes the python code for the instancequeries. So far, it replaces it by 'NotImplemented'exceptions. In a later patch of this series, theremaining part is remove completely.
Make watcher use queries properly
The watcher did not yet use the query client to makequeries. This patch fixes that.
Adjust console behavior wrt to operation state
Unlike its cli counterpart, the rapi console operation didnot properly take into account whether the instance isactually running. This patch fixes this problem.
User query client necessary for instance queries
There were some places left when code querying for instanceinformation did not use the query client yet.
Switch to Haskell for group queries
This patch removes the group query implementationin python in order to use the new Haskell implementation.
Use query client when neccessary for group queries
This patch makes code use the query client for groupqueries where necessary in order to remove the pythonqueries soon.
Switch to haskell for export (aka backup) queries
This patch removes the python implementation of export(aka backup) queries. So far, it is replaced by'NotImplemented' exceptions, but later in this seriesit will be replaced completely.
Switch to Haskell for network queries
This patch removes the python implementation of networkqueries and replaces it with 'NotImplemented' exceptions.It will be removed completely once all queries areswitched to Haskell.
Disable node query code
This patch removes the python query implementation fornodes. So far, the code is replaced by 'NotImplemented'exceptions, because the overall structure of query classescan be removed more easily at once when all queryimplementations are ready to be removed....
masterd: implement query via luxi
The master daemon so far still did queries via the pythonimplementation. This patch implements that it uses thehaskell implementation and removes the node queries fromthe list of OP-queriable entities.
Make watcher use query clients
So far, the watcher was still issuing node queriesvia the python implementation. This patche switches tothe haskell implementation.
Move GetClient() to runtime module
Soon both, the cli and the masterd, will use a luxiclient to run queries. The method to obtain and configurethe luxi client with the right address can be reused fromcli to masterd here. Therefore, this patch moves the...
Implement 'QueryInstances' call in Haskell luxi server
While the command line uses the generic 'Query' call,rapi calls 'QueryInstances'. 'QueryInstances' so farwas not fully implemented in the Haskell implementationof the luxi server. This was discovered when trying to...
Fix bug regarding node UUID in haskell node queries
When moving from python to haskell node queries, a bugwas discovered where a node's UUID was mistakenlycompared to a node's name. This indirectly caused thecluster epo operation to fail, because it was not...
Use a query client for node queries where necessary
To remove the node query's python implementation soon, thispatch makes all users of the queries call the haskellimplementation instead of the python one.
Remove --enable-split-query option
Switching from python to haskell queries, this patchremoves the option to dis/enable the haskell queriesat configure time.
Add Xen paused state as a Ganeti running state
The QA test 'TestClusterBurnin' pauses instances and the Xen statedomain 'paused' was not being handled properly by Ganeti because itwas not being classified as either a running or shutdown state. Thispatch adds the 'paused' state to the set of running states by...
Fix wrong expectation in hsqueeze shelltest
The shelltests expected the wrong position of the -f optionas well. Fix this.
hsqueeze: fix position of option in gnt-node power
hsqueeze can produce a shell script with the commands to squeezethe cluster; in the script, fix the position of the '-f' optionin the 'gnt-node power' command.
build_chroot: hard-code the version of blaze-builder
The newest version does not build on debian squeeze, so avoidit beeing pulled in as a dependency.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Jose Lopes <jabolopes@google.com>
Fix broken TAGS generation
This file is used for development to point to code definitions. Apparently thishas been broken since Februrary [cc40185]. The improved version fixes theproblem and uses Exuberant Ctags <http://ctags.sourceforge.net/>. It uses...
Add patching QA configuration files on buildbots
Add missing spindles paramter to idisk
When spindles where added to Ganeti, apparently it was forgottento add the parameter to the Haskell data type as well. Do this now.
Parse spindles as integer
The disk parameter "spindles", if present, has to be an integer.Enforce this already when parsing the command line, instead ofpassing around the string representation of that integer.
Add an IPv6 check for the QA test of custom SSH ports
If any of the nodes involved would be reached through IPv6, the test isskipped. This is because the required testing setup using "iptables" issuiteble only for IPv4 and doesn't work for IPv6 connections....
hsqueeze: describe option -C in the man page
The option -C was recently added to hsqueeze, but not documented inthe man page. Add the documentation now.
Allow the NIC VLAN to be set to an empty string
The NIC VLAN has previously not been modified via Haskell, causing theINicParams class not to be used. With the recent job queuerefactorings, a modification definition is recorded, and for an emptystring (which is a legal default value) a crash happens. This patch...
Add the aggregate NIC VLAN instance field
Allow the retrieval of the VLANs of all the NICs through nic.vlans.
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Add NIC VLAN field retrieval to Haskell queries
The field was added to Python queries in an earlier version, and nowhas to be added to the Haskell queries as well.
Show running times of whole test blocks
Similarly to simple tests, test blocks are now demarcated (in boldface)and measured.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Add more color options to QA
Allow combination of colors, background colors, bold, reverse video,etc.
Add more thorough check for terminal color support to QA
It uses `tput` to check if the current terminal has enough colors.
Add tests for hsqueeze commands
In the examples, pick a node that certainly has to be powered on or off,respectively, and verify that the power command is there, as wellan appropriate migration command in the right position.
hsqueeze: add option to show or save commands
Add an option to hsqueeze to show, or save in a file, the commandsthat have to be carried out.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
hsqueeze: when balancing also keep the move sequence
In hsqueeze, when computing the balancing sequence, alsoremember the sequence of moves that lead there.
Add function to get the moves between two configurations
Add a function that, given two adjacent cluster configurations ofa balancing sequence, computes the moves that led from the firstto the second configuration.
In the list of involved nodes, drop "no secondary"
When grouping moves into jobs, a new job set is started, if the newmove involves a node also touched by a previous move. When computingthe list of involved nodes, the new primary and secondary nodes of the...
Move saving of a command list to CLI
Move the function that saves a list of a command in a fileto CLI.hs. In this way, it is reusable by other htools.
Merge branch 'stable-2.10' into master
Merge branch 'stable-2.9' into stable-2.10
Add NodeGroup to InstanceConsoleInfoParams
Before, calls to `gnt-instance list -o console` with an instance on anode with a custom SSH port failed because of missing groupconfiguration. This patch fixes the problem.
Signed-off-by: Petr Pudlak <pudlak@google.com>...
Add a simple QA test for a group with a custom SSH port
The test is turned on by setting `"group-custom-ssh-port": true` inthe config.
Unlike `"ssh-port": ...` in "options", this is a single test thatdoesn't affect the rest of the tests.
Export `NodeAdd` and `NodeRemove` from qa_node
This is needed for testing SSH ports so that we can remove a node froma group, add it to a group with a custom SSH port, and then undo thechange.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Allow QA node acquisition and group creation using `with`
For acquiring nodes use `with AcquireManyNodes(num): ...`. The nodeswill be released automatically.For creating a new group use `with NewGroupCtx() as group: ...`. Thegroup will be removed automatically....
Update NEWS with information about custom SSH ports
Update custom SSH ports config when downgrading to 2.10
If SSH ports other than 22 are configured, abort the config downgrade,as the operation would certainly fail. Otherwise just removendparams/ssh_port from the configuration.
Include ssh_port in the man page
Added to the other node parameters in ganeti.rst.
Update QA to test custom SSH ports
This patch adds the `ssh-port` option. If set to a non-standard port,the QA script sets up the default node group with this port, and beforerunning tests it adds `iptable` rules to all nodes so that the nodessee each other's SSH servers as running on this port....
Use custom SSH ports for other operations
In particular copyfile and renew-crypto.
Use configured SSH ports when connecting to a console
This is accomplished by passing the corresponding node group tohv_*.py. Tests for hv_*.py that call GetInstanceConsole updated.
Update unit tests for custom SSH ports
The unit tests now include the configurations of SSH ports for nodegroups/nodes.
Use custom SSH ports in node groups when working with nodes
Calling `gnt-instance console` with a custom SSH port doesn't work yet.
Add "ndp/ssh_port" node group configuration parameter
The parameter is added to Haskell sources, from which the correspondingPython code is generated.
Add a method retrieving `NodeGroup` objects by group name
Also add a method for the retrieval of "ndparams" (node parameters)of groups, not just nodes.
group queries: test niceSort and remove FIXME
In an effort to get rid of the python queries soon, thispatch fixes a FIXME of the group queries regarding themissing testing of niceSort in this context. Due to thelack of actually weirdly named hostnames, this patch...
Don't allow optional node parameters
Ganeti does not support optional fields in parameters(hypervisor-params, disk-params, etc.). OpenVSwitch related nodeparameters were the exception to this rule, which caused numerousproblems related to import/export and (de-)serialization....
Move OVS node parameters to the right place
OpenVSwitch node parameters were passed as separate command line flags.This was not only inconsistent with all other parameters(hypervisor-params, disk-params, ...), but also with the documentedbehavior in the man pages....
Make NIC VLAN queryable
Add an additional query field which makes it possible to query forVLANs of instance NICs.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Pass VLAN parameter correctly in moves
Fix the assignment of a wrong value to the VLAN and name parameters ofNICs during instance moves.
Use constant instead of raw string
Use the appropriate constant instead of a raw string.
Fix instance info
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Add QA tests for instance shutdown detection
Add QA tests for instance shutdown detection with Xen PVM and HVMhypervisors.
Fix class methods in KVM hypervisor
Several methods in KVM hypervisor behave like 'classmethods', however,they did not have the '@classmethod' annotation. This patch adds'@classmethod' annotations to several of these methods with the goalof making '_StopInstance' available to the QA. This step is necessary...
Update tests
Update hypervisor unit tests.
Watcher cleans user shutdown domains
Update watcher to destroy shutdown domains when it detects that adomain is in the state 'USER_down'. This requires adding logicsimilar to that of restart instance, that is tracking the number ofcleanup attempts in the watcher's state and give up if a given number...
Haskell instance queries report 'USER_down'
Update instance queries on the Haskell codebase to report 'USER_down',similarly to the Python instance queries.
Python instance queries report 'USER_down'
Update instance queries on the Python codebase to report 'USER_down'in the situation where the user has shutdown a Xen domain but Ganetistill marks this domain as 'ADMIN_up' in its configuration file.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>...
Prevent instance start when user down
Prevent starting a given instance when that instance has been shutdownby the user, given that the instance must be first properly shutdown.
Xen handle domain shutdown
Update Xen backend to properly recognize when a domain has beenshutdown by the user and to properly cleanup a shutdown domain whenGaneti requests Xen to stop this domain.
Add instance state 'USER_down'
Add instance state 'USER_down' which is a state used in reporting onlyand it represents the situation in which the user has shutdown theinstance but Ganeti's configuration still has this instance marked as'ADMIN_up'.
Preserve Xen domain after shutdown
Change Xen domain configuration to preserve the domain state aftershutdown, as opposed to automatic reboot, which is the defaultbehaviour.
Add Haskell hypervisor instance state
Add 'InstanceState' datatype which is the Haskell counterpart of thePython type 'HvInstanceState'.
Add Python hypervisor instance state
Add 'HvInstanceState' which represents the state of an instance(either 'running' or 'shutdown') and fix the remaining hypervisorbackends to return a value of this type. Before this patch, eachbackend was returning their own value, some were returning strings,...
Fix typo in documentation
Merge 'hs2py-constants' into 'hs2py'
Merge code of 'hs2py-constants' into 'hs2py', which requires addingflag '--constants', and merge target 'src/hs2py-constants' into'src/hs2py' in 'Makefile.am', also updating dependencies andvariables.