History | View | Annotate | Download (26.1 kB)
Allow rpc.MakeLegacyNodeInfo to parse non-LVM results
'MakeLegacyNodeInfo' is not the best place for this, but we'd have toduplicate it if we wanted a LVM-less version, so the easiest is to addan optional parameter that allows it to accept/skip LVM-less results....
Call node_info RPCs with the exclusive_storage flag
The flag is read from the configuration and passed to the RPC.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fixes and improvements to comments
Some fixes, added more information in a few points, removed a stale (5+year old) TODO comment.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Pass netinfo in rpcs
If a nic has a network field then encapsulate a network object innetinfo slot for every rpc. This is needed to pass network info toscripts managing nics (kvm-vif-bridge).
Introduce _BuildNetworkEnv().
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>...
Merge branch 'devel-2.6'
Implement virtual cluster support in Python code
- pathutils: Prepend node-specific prefix path- RPC: Use virtual paths (see vcluster.py)- SSH: Pass environment variables, use destination's node directory when copying files using scp, use GANETI_HOSTNAME to determine hostname...
Migrate lib/rpc.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
rpc: Remove duplicated logic, fix unittests
Commit 5fce6a89 changed RpcRunner._InstDict to add the disk parameterson all encoded instances. It didn't remove a special case in“_InstDictOspDp”. Update and fix unittests as well.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
objects, rpc: Code cleanup
objects.Disk.ComputeLDParams: Simplify structure, don't use andoverwrite local variables.
rpc.AnnotateDiskParams: Replace explicit loop.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Annotate disk params on instance_start
We call _GatherAndLinkBlockDevs during the process, which in turncalls _RecursiveFindBD. This needs disk parameters to work.
See also commit b8291e0.
This was reported by Ansgar and Damien.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>...
Move _MakeLegacyNodeInfo to rpc
This function will also be needed for the iallocator refactor. Also thisfunction does not belong to cmdlib but rather to rpc. Lets move itthere.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Move necessary constants from rpc & rpc_defs to constants
As timeout constants will now be used also on Haskell side, they havebeen moved from rpc_defs and rpc to constants. The same is true forconnection timeout for connecting to nodes.
Signed-off-by: Agata Murawska <agatamurawska@google.com>...
Annotate disk params on instance_os_add
We call _OpenRealBD during the process and this needs disk parameters towork. This was reported by Constantinos.
The fix is very ugly though.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
rpc: Adding new encoders for annotating disk parameters
rpc: Adding helper to annotate disk params
RPC: Add a new client type for DNS only
This patch moves the “call_version” to a new RPC client definition andthen adds a new runner using the DNS resolver for getting the hostaddress.
The standard “BootstrapRunner”, where the call was before, tries to...
Replace single- with double-quotes
In at least two cases "%s" is replaced with str(), too.
Accept both PUT and POST in noded
Currently, noded requires PUT, even though the semantics of the RPCcalls do not match a PUT. We change the code accept both PUT and POST,with the intention to remove the PUT support in a later version.
On the client side, the RPC code is changed to send POST requests....
rpc.ConfigRunner: Fix uploading file
Commit 601dfcb made some changes to the default encoders. This made“ConfigRunner.call_upload_file” fail due to a missing encoder. Thispatch applies the necessary changes to ConfigRunner and updates theunittest for “call_upload_file”....
rpc._PrepareFileUpload: Use FileStatHelper
Use fstat(2) on the file handle instead of doing a separate stat(2).
rpc: Change signature of RpcRunner
Instead of receiving the whole masterd context, it only gets theconfiguration and the lock monitor callback. This simplifiesunittesting.
Additionaly, two new arguments for testing are added.
Add unittest for uploading file via RPC
After some preparing patches this unittest can finally be added.
rpc._RpcClientBase: Add check for number of arguments
Just an additional check, useful in unittests.
Add unittests for RPC client
This patch adds a number of unittests for the RPC client base class.Some small changes were necessary in “rpc.py” to allow for bettertesting.
rpc: Style fix
“assert” is a statement, not a function.
rpc._NodeConfigResolver: Support resolving offline nodes
This is needed to powercycle a node marked offline.
rpc._RpcProcessor: Make read timeout a non-keyword argument
It is always used and can be set to “None” if intended.
rpc: Add support for resolver options callback
This adds support for a callback returning name resolver options. Thisis required for powercycling offline nodes. While it would be possibleto implement this using a separate RPC client class, doing so would...
rpc: Pass resolver options to actual resolver
Extend RPC definition change with another parameter
This will be used in a later patch for parameter pre-processing. Fornow we just add the parameter as 'None' and add a consistency checkfor the parameter definitions.
Signed-off-by: Iustin Pop <iustin@google.com>...
Change internal RPC client body values
Currently, all RPC payloads sent by the client to the remote nodedaemons must be identical, due to how the data is passedinternally. This is deficient in both use (from the programmer's pointof view) and from the network traffic (cluster verify/disk data...
serializer: Remove JSON indentation and dict key sorting
Serializing to JSON using “simplejson” is significantly slower whenindentation and/or sorting of dictionary keys is used. In simplejson 1.xthe difference isn't that big, but with simplejson 2.x the difference...
rpc: Call result processor once for each node result
… instead of calling it with the whole results dictionary. Thisfixes an issue when replacing disks (and all other cases whereresult processors are used).
RPC/test_delay: Use callable for timeout calculation
This avoids having to override the function in the RPC runner.
rpc: Move post-processor functions into definitions file
This way the generated code no longer contains arbitrary code.Post-processing functions are used by reference.
rpc: Use definitions directly instead of via generated code
Until now “autotools/build-rpc” would read the definition of all RPCsand write them to a new file, “lib/_generated_rpc.py” with somemodifications. With this patch the generated code loads the definitions...
rpc: Make “test_delay” RPC actually work
Until now it would just call itself, eventually failing.
Show RPC calls from config in lock monitor
With this patch all RPC calls at runtime of masterd will show up in thelock monitor. There is a chicken-and-egg issue with initializing theconfiguration with a context since the lock manager, containing themonitor, requires the configuration. This is worked around by setting...
rpc: Improve argument definitions
Instead of in-place code now each argument has an (optional) kindassigned. The RPC client will then take care of encoding each argumentaccording to its definition. A new base (_RpcClientBase) is added totake care of encoding and dispatching calls....
rpc: Change {import,export}_start to take source/dest in single argument
This simplifies the RPC argument encoding code.
Add missing get_migration_status RPC definition
Add the instance_get_migration_status RPC definition and the neededhelper.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
rpc: Remove leftovers from timeout table
rpc: Remove _Static* and other unused functions
These are no longer needed.
rpc: Convert last two calls to generated code
These two calls, “upload_file” and “write_ssconf_files” are treatedseparated as they're used by the configuration, where we can't use thenormal resolver.
There's still some duplicated code in rpc.py, but that will be taken...
rpc: Convert call for HV parameter validation
Instead of filling the parameters in the RPC layer, that is now donebefore the wrapper is called, thereby simplifying the wrapper.
rpc: Convert two more instance-specific calls
Interface changes were necessary as these took more parameters than wereactually passed over the wire. Those parameters were just passed to theinstance serialization function.
Move bootstrap-related RPC to generated wrappers
With this patch, only 6 RPC are left as old-style code.
Convert job queue's RPC to generated code
With these changes job queue RPC will finally show up on the lockmonitor. See below for an example. A job queue-specific class is used torestrict the use of a static list for name resolution to the job queue.Further improvements can be made to not re-create the whole RPC client...
rpc: Convert wrappers for starting import/export daemons
rpc: Convert storage-related calls
rpc: Convert blockdev-related calls
rpc: Convert instance-related calls
rpc: Convert misc calls
rpc: Convert file-storage-related calls
rpc: Convert node-related calls
rpc: Convert OS-related calls
rpc: Convert X509 calls
rpc: Convert import/export functions
rpc: Add helpers
These will be used by the generated RPC wrappers.
rpc: Add support for generated RPC wrappers
rpc: Disable timeout check
This is in preparation for the definition-based RPC model, wherethe timeout will be stored within the definition.
Fix parameters to RPC "blockdev_rename"
Before this patch it would take a variable number of arguments, whichisn't very good for a definition-based approach.
Fix parameters to RPC "os_validate"
All other RPC wrappers take the node name(s) as the first parameter.
Remove “node_tcp_ping” RPC
It is not used anywhere.
http.client: Remove HTTP client pool code
This patch removes all remains of the HTTP client pool. Newly added unittestsprovide 96% coverage on http.client.
rpc: Remove thread-local storage with HTTP pool
The HTTP pool is no longer used.
Merge branch 'devel-2.5'
Merge branch 'stable-2.5' into devel-2.5
Merge branch 'devel-2.4' into stable-2.5
rpc: Disable HTTP client pool and reduce memory consumption
We noticed that “ganeti-masterd” can use large amounts of memory,especially on large clusters. Measurements showed a single PycURL clientusing about 500 kB of heap memory (the actual usage depends on versions,...
Add cluster netmask parameter
Add the master_netmask cluster parameter, that represents the netmask ofthe master IP, encoded as a CIDR suffix.
This parameter can be set via the --master-netmask of gnt-cluster initand gnt-cluster modify. The default behaviour is to be consistent with...
Split starting and stopping master IP and daemons
Make migration RPC non-blocking
To add status reporting for the KVM migration, the instance_migrate RPCmust be non-blocking. Moreover, there must be a way to represent themigration status and a way to fetch it.
http.client: Add nice name to requests
With this change a node name instead of the IP address can be shown forpending RPC requests:Name Pendingrpc/node18.example.com/test_delay thread:Jq1/Job692/TEST_DELAY
rpc/http: Show pending RPC requests in lock monitor
Not all requests use an instance of RpcRunner yet and therefore won'tshow up (only instances have access to the global Ganeti context).Currently only the IP address is accessible. Another patch will add a...
rpc: Overhaul client structure
- Clearly separate node name to IP address resolution into separate functions- Simplified code structure (one code path instead of several)- Fully unittested- Preparation for more RPC improvements
rpc: Make compression function module-global
No need to keep it in the class.
Keep only one global RPC runner in Ganeti context
Instead of having one RPC runner per mcpu processor this will keep onlyone instance as part of the masterd-wide Ganeti context. Upcomingpatches will change the RPC runner to report pending requests to the...
DeprecationWarning fixes for pylint
In version 0.21, pylint unified all the disable-* (and enable-*)directives to disable (resp. enable). This leads to a lot ofDeprecationWarning being emitted even if one uses the recommendedversion of pylint (0.21.1, as stated in devnotes.rst)....
Instance transfer: export component name to backend
This modifies the RPC layer to export the component name too to thebackend, so that it can be used in log files and messages.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add gnt-instance start --pause
Creates the instance, but pauses execution before booting. This combinedwith 'gnt-instance console' unpausing instances means that the entireboot process can be viewed and monitored.
Signed-off-by: Stephen Shirley <diamond@google.com>...
Merge branch 'devel-2.4'
RPC/Backend: Make UploadFile uid and gid agnostic
Implement grow dry-run at RPC level
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add bdev_sizes RPC call
The bdev_sizes multi-node RPC call returns the sizes of the requestedblock devices on the desired nodes. Its intended use is to verify theexistence of a block device on a given node for shared block storagesupport.
Block device paths are expected to lie under constants.BLOCKDEV_DIR...
Fix RPC mismatch in blockdev_getsize[s]
Commit 92fd2250 added consistency checks in the RPC layer, which brokethe call_blockdev_getsizes RPC call (declared with 's' at the end inrpc.py, without 's' in the node daemon).
The immediate fix is to correct the rpc function name, the long term...
Re-create instance disk symlinks on activate
This patch implements recreation of instance disk symlinks when theactivate-disks operation is run. Until now, it was not possible tore-create these symlinks without stopping and starting or migrating aninstance as the RPC call where this is done was in instance startup...
Add RPC calls for pause/resume sync of devices
RPC: mark jobqueue functions as URGENT
Recently, we've seen more and more cases of a specific breakagepattern in Ganeti: master candidates which are semi-alive (as in, theyrespond to ping, they can complete a TCP/SSL handshake, but otherwisethe root filesystem is broken) cause lots of confusion within masterd....
RpcResult: simplify some asserts
data ≫ code, eom.
Adding missing timeout parameter to rpc interface
Adding RPC infrastructure to call oob
Fix disk checks in “gnt-cluster verify”
Tests have shown that the changes in commit b8d26c6e5 don't work aswanted. If any disk wasn't found on the node, all disks located on thesame node would show as faulty. The cause was incorrect exceptionhandling on the node....
LUClusterVerify: Complain if disk is marked faulty
This will show a warning if, for example, one side of a DRBDdisk becomes unavailable. The data is collected separatelyfrom the other verification data.
Example output:
rpc: Work around epydoc warning
Aliasing the “threading” module allows us to avoid the “No informationavailable for ganeti.rpc._RpcThreadLocal's base threading.local” warningby epydoc.
Adding RPC call for blockdev_wipe
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
gnt-instance reinstall: Allow overriding OS parameters
This allows OS installation scripts to make use of special parameters,e.g. to retain some data on reinstallation.
The RAPI resource is not updated as it takes all parameters via thequery string and encoding arbitrary data in a query string is tricky....
Merge branch 'devel-2.2'
Conflicts: lib/rpc.py (trivial, copyright header)
RPC: disable curl's Expect header
This patch solves the very slow (~8-9 seconds) gnt-instance modifybehaviour. Well, it solves in general the slow RPC behaviour, but it wasmost visible in that LU.
It seems that curl's behaviour with regard to file uploads (via PUT) and...