History | View | Annotate | Download (24.1 kB)
rpc: Adding new encoders for annotating disk parameters
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
rpc: Adding helper to annotate disk params
RPC: Add a new client type for DNS only
This patch moves the “call_version” to a new RPC client definition andthen adds a new runner using the DNS resolver for getting the hostaddress.
The standard “BootstrapRunner”, where the call was before, tries to...
Replace single- with double-quotes
In at least two cases "%s" is replaced with str(), too.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Accept both PUT and POST in noded
Currently, noded requires PUT, even though the semantics of the RPCcalls do not match a PUT. We change the code accept both PUT and POST,with the intention to remove the PUT support in a later version.
On the client side, the RPC code is changed to send POST requests....
rpc.ConfigRunner: Fix uploading file
Commit 601dfcb made some changes to the default encoders. This made“ConfigRunner.call_upload_file” fail due to a missing encoder. Thispatch applies the necessary changes to ConfigRunner and updates theunittest for “call_upload_file”....
rpc._PrepareFileUpload: Use FileStatHelper
Use fstat(2) on the file handle instead of doing a separate stat(2).
rpc: Change signature of RpcRunner
Instead of receiving the whole masterd context, it only gets theconfiguration and the lock monitor callback. This simplifiesunittesting.
Additionaly, two new arguments for testing are added.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Add unittest for uploading file via RPC
After some preparing patches this unittest can finally be added.
rpc._RpcClientBase: Add check for number of arguments
Just an additional check, useful in unittests.
Add unittests for RPC client
This patch adds a number of unittests for the RPC client base class.Some small changes were necessary in “rpc.py” to allow for bettertesting.
rpc: Style fix
“assert” is a statement, not a function.
rpc._NodeConfigResolver: Support resolving offline nodes
This is needed to powercycle a node marked offline.
rpc._RpcProcessor: Make read timeout a non-keyword argument
It is always used and can be set to “None” if intended.
rpc: Add support for resolver options callback
This adds support for a callback returning name resolver options. Thisis required for powercycling offline nodes. While it would be possibleto implement this using a separate RPC client class, doing so would...
rpc: Pass resolver options to actual resolver
Extend RPC definition change with another parameter
This will be used in a later patch for parameter pre-processing. Fornow we just add the parameter as 'None' and add a consistency checkfor the parameter definitions.
Signed-off-by: Iustin Pop <iustin@google.com>...
Change internal RPC client body values
Currently, all RPC payloads sent by the client to the remote nodedaemons must be identical, due to how the data is passedinternally. This is deficient in both use (from the programmer's pointof view) and from the network traffic (cluster verify/disk data...
serializer: Remove JSON indentation and dict key sorting
Serializing to JSON using “simplejson” is significantly slower whenindentation and/or sorting of dictionary keys is used. In simplejson 1.xthe difference isn't that big, but with simplejson 2.x the difference...
rpc: Call result processor once for each node result
… instead of calling it with the whole results dictionary. Thisfixes an issue when replacing disks (and all other cases whereresult processors are used).
RPC/test_delay: Use callable for timeout calculation
This avoids having to override the function in the RPC runner.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
rpc: Move post-processor functions into definitions file
This way the generated code no longer contains arbitrary code.Post-processing functions are used by reference.
rpc: Use definitions directly instead of via generated code
Until now “autotools/build-rpc” would read the definition of all RPCsand write them to a new file, “lib/_generated_rpc.py” with somemodifications. With this patch the generated code loads the definitions...
rpc: Make “test_delay” RPC actually work
Until now it would just call itself, eventually failing.
Show RPC calls from config in lock monitor
With this patch all RPC calls at runtime of masterd will show up in thelock monitor. There is a chicken-and-egg issue with initializing theconfiguration with a context since the lock manager, containing themonitor, requires the configuration. This is worked around by setting...
rpc: Improve argument definitions
Instead of in-place code now each argument has an (optional) kindassigned. The RPC client will then take care of encoding each argumentaccording to its definition. A new base (_RpcClientBase) is added totake care of encoding and dispatching calls....
rpc: Change {import,export}_start to take source/dest in single argument
This simplifies the RPC argument encoding code.
Add missing get_migration_status RPC definition
Add the instance_get_migration_status RPC definition and the neededhelper.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
rpc: Remove leftovers from timeout table
rpc: Remove _Static* and other unused functions
These are no longer needed.
rpc: Convert last two calls to generated code
These two calls, “upload_file” and “write_ssconf_files” are treatedseparated as they're used by the configuration, where we can't use thenormal resolver.
There's still some duplicated code in rpc.py, but that will be taken...
rpc: Convert call for HV parameter validation
Instead of filling the parameters in the RPC layer, that is now donebefore the wrapper is called, thereby simplifying the wrapper.
rpc: Convert two more instance-specific calls
Interface changes were necessary as these took more parameters than wereactually passed over the wire. Those parameters were just passed to theinstance serialization function.
Move bootstrap-related RPC to generated wrappers
With this patch, only 6 RPC are left as old-style code.
Convert job queue's RPC to generated code
With these changes job queue RPC will finally show up on the lockmonitor. See below for an example. A job queue-specific class is used torestrict the use of a static list for name resolution to the job queue.Further improvements can be made to not re-create the whole RPC client...
rpc: Convert wrappers for starting import/export daemons
rpc: Convert storage-related calls
rpc: Convert blockdev-related calls
rpc: Convert instance-related calls
rpc: Convert misc calls
rpc: Convert file-storage-related calls
rpc: Convert node-related calls
rpc: Convert OS-related calls
rpc: Convert X509 calls
rpc: Convert import/export functions
rpc: Add helpers
These will be used by the generated RPC wrappers.
rpc: Add support for generated RPC wrappers
rpc: Disable timeout check
This is in preparation for the definition-based RPC model, wherethe timeout will be stored within the definition.
Fix parameters to RPC "blockdev_rename"
Before this patch it would take a variable number of arguments, whichisn't very good for a definition-based approach.
Fix parameters to RPC "os_validate"
All other RPC wrappers take the node name(s) as the first parameter.
Remove “node_tcp_ping” RPC
It is not used anywhere.
http.client: Remove HTTP client pool code
This patch removes all remains of the HTTP client pool. Newly added unittestsprovide 96% coverage on http.client.
rpc: Remove thread-local storage with HTTP pool
The HTTP pool is no longer used.
Merge branch 'devel-2.5'
Merge branch 'stable-2.5' into devel-2.5
Merge branch 'devel-2.4' into stable-2.5
rpc: Disable HTTP client pool and reduce memory consumption
We noticed that “ganeti-masterd” can use large amounts of memory,especially on large clusters. Measurements showed a single PycURL clientusing about 500 kB of heap memory (the actual usage depends on versions,...
Add cluster netmask parameter
Add the master_netmask cluster parameter, that represents the netmask ofthe master IP, encoded as a CIDR suffix.
This parameter can be set via the --master-netmask of gnt-cluster initand gnt-cluster modify. The default behaviour is to be consistent with...
Split starting and stopping master IP and daemons
Make migration RPC non-blocking
To add status reporting for the KVM migration, the instance_migrate RPCmust be non-blocking. Moreover, there must be a way to represent themigration status and a way to fetch it.
rpc/http: Show pending RPC requests in lock monitor
Not all requests use an instance of RpcRunner yet and therefore won'tshow up (only instances have access to the global Ganeti context).Currently only the IP address is accessible. Another patch will add a...
http.client: Add nice name to requests
With this change a node name instead of the IP address can be shown forpending RPC requests:Name Pendingrpc/node18.example.com/test_delay thread:Jq1/Job692/TEST_DELAY
rpc: Overhaul client structure
- Clearly separate node name to IP address resolution into separate functions- Simplified code structure (one code path instead of several)- Fully unittested- Preparation for more RPC improvements
rpc: Make compression function module-global
No need to keep it in the class.
Keep only one global RPC runner in Ganeti context
Instead of having one RPC runner per mcpu processor this will keep onlyone instance as part of the masterd-wide Ganeti context. Upcomingpatches will change the RPC runner to report pending requests to the...
DeprecationWarning fixes for pylint
In version 0.21, pylint unified all the disable-* (and enable-*)directives to disable (resp. enable). This leads to a lot ofDeprecationWarning being emitted even if one uses the recommendedversion of pylint (0.21.1, as stated in devnotes.rst)....
Instance transfer: export component name to backend
This modifies the RPC layer to export the component name too to thebackend, so that it can be used in log files and messages.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add gnt-instance start --pause
Creates the instance, but pauses execution before booting. This combinedwith 'gnt-instance console' unpausing instances means that the entireboot process can be viewed and monitored.
Signed-off-by: Stephen Shirley <diamond@google.com>...
Merge branch 'devel-2.4'
RPC/Backend: Make UploadFile uid and gid agnostic
Implement grow dry-run at RPC level
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add bdev_sizes RPC call
The bdev_sizes multi-node RPC call returns the sizes of the requestedblock devices on the desired nodes. Its intended use is to verify theexistence of a block device on a given node for shared block storagesupport.
Block device paths are expected to lie under constants.BLOCKDEV_DIR...
Fix RPC mismatch in blockdev_getsize[s]
Commit 92fd2250 added consistency checks in the RPC layer, which brokethe call_blockdev_getsizes RPC call (declared with 's' at the end inrpc.py, without 's' in the node daemon).
The immediate fix is to correct the rpc function name, the long term...
Re-create instance disk symlinks on activate
This patch implements recreation of instance disk symlinks when theactivate-disks operation is run. Until now, it was not possible tore-create these symlinks without stopping and starting or migrating aninstance as the RPC call where this is done was in instance startup...
Add RPC calls for pause/resume sync of devices
RPC: mark jobqueue functions as URGENT
Recently, we've seen more and more cases of a specific breakagepattern in Ganeti: master candidates which are semi-alive (as in, theyrespond to ping, they can complete a TCP/SSL handshake, but otherwisethe root filesystem is broken) cause lots of confusion within masterd....
RpcResult: simplify some asserts
data ≫ code, eom.
Adding missing timeout parameter to rpc interface
Adding RPC infrastructure to call oob
Fix disk checks in “gnt-cluster verify”
Tests have shown that the changes in commit b8d26c6e5 don't work aswanted. If any disk wasn't found on the node, all disks located on thesame node would show as faulty. The cause was incorrect exceptionhandling on the node....
LUClusterVerify: Complain if disk is marked faulty
This will show a warning if, for example, one side of a DRBDdisk becomes unavailable. The data is collected separatelyfrom the other verification data.
Example output:
rpc: Work around epydoc warning
Aliasing the “threading” module allows us to avoid the “No informationavailable for ganeti.rpc._RpcThreadLocal's base threading.local” warningby epydoc.
Adding RPC call for blockdev_wipe
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
gnt-instance reinstall: Allow overriding OS parameters
This allows OS installation scripts to make use of special parameters,e.g. to retain some data on reinstallation.
The RAPI resource is not updated as it takes all parameters via thequery string and encoding arbitrary data in a query string is tricky....
Merge branch 'devel-2.2'
Conflicts: lib/rpc.py (trivial, copyright header)
RPC: disable curl's Expect header
This patch solves the very slow (~8-9 seconds) gnt-instance modifybehaviour. Well, it solves in general the slow RPC behaviour, but it wasmost visible in that LU.
It seems that curl's behaviour with regard to file uploads (via PUT) and...
Add RPC calls to update /etc/hosts
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix a few epydoc docstrings
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Removing all ssh setup code from the core
Support IPv6 node add
Support for resolving hostnames to IPv6 addresses
This patch enables IPv6 name resolution by using socket.getaddrinfoinstead of socket.gethostbyname_ex.
It renames the HostInfo class to Hostname and unifies its use throughoutthe code. This is achieved by using static calls where no object is...
Always use address instead of hostname in rpc.Client
In light of the upcoming IPv6 support, this patch enables the rpc.Clientto always use a node's address to connect to it. This is necessary as wedo not want to rely on name resolution to connect to the correct IP...
Move the UsesRPC decorator from cli to rpc
This is needed because not just the cli scripts need this decorator, butthe master daemon too (and it already duplicated the code once).
In cli.py we just leave a stub, so that we don't have to modify all thescripts to import rpc.py....
Convert RPC client to PycURL
Instead of using our custom HTTP client, using PycURL's multiinterface allows us to get rid of the HTTP client threadpool.The majority of the code is still in the ganeti.http.clientmodule.
A simple per-thread HTTP client pool gives cURL a chance to...
Introduce lib/netutils.py
This patch moves network utility functions to a dedicated module.
Add drbd_helper rpc call
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add OS parameters to cluster and instance objects
The patch also modifies the instance RPC calls to fill the osparameterscorrectly with the cluster defaults, and exports the OS parameters inthe instance/OS environment.
Introduce an RPC call for OS parameters validation
While we only support the 'parameters' check today, the RPC call isgeneric enough that will be able to support other checks in the future.The backend function will both validate the parameters list (so as to...
Merge branch 'devel-2.1' into master
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>