Export instance disk spindles
Instance disk spindles can be queried via LUXI and RAPI, and are exportedthrough the allocator interface. This is a prerequisite for htools tohandle spindles.
The length of a RAPI query including all the instance fields now has become...
Remove some unused Python code
This patch removes code which is no longer used due to refactoring:
- http.InitSsl, last usage removed in commit 33231500 (“Convert RPC client to PycURL”)- rapi.baserlib.MakeParamsDict, last usage remove in commit 4e5a68f8...
Warn on invalid lines in HTTP user files
Without this change, invalid lines or values would be silently ignored.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Merge branch 'devel-2.6'
http/__init__.py: Remove extraneous argument
pylint complained, I fixed it, and unfortunately pushed too early.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
http: Add wrapper for mimetools.Message
A newly added piece of code will also have to parse headers, so havingthis wrapper saves us from copying this part of code.
Conflicts: lib/hypervisor/hv_xen.py: trivial
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Disable E1101 on ganeti/http/server.py:424
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Factorize removing comments and empty lines from string
This will also be used for verifying the file storage directory.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Bump pep8 version to 1.2
Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:
- bump version in the docs- silence some new checks that are wrong due to our indent=2 instead of 4- fix lots of errors in the code where the indentation was wrong by 1...
errors: Add exception for RAPI testing utilities
This exception is raised to abort before actually sending a LUXI call(there is no LUXI server involved in the test). The testing utilitiescatch the exception to report a success (i.e. the code didn't throw...
http.server: Factorize request handling even more
This splits even more parts of the request handling code into a separateclass. Doing so allows us to reuse this part of the code for tests (e.g.mocks). Unlike before now the error handling can also be reused....
http.server: Move error message formatting to handler class
Like before this patch moves more functionality from the actual serverclass into a separate handler class. At the same time the function ischanged to return both content-type and body instead of relying on a...
Split handling HTTP requests into separate class
Until now HTTP requests were handled in the same class as incomingconnections (http.server.HttpServer). With this change the requesthandling is delegated to a separate class which can be re-used in tests...
http.server: Factorize request handling
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
http.client: Remove use of PycURL's “reset” function
We don't re-use cURL objects anymore, so there's no need to reset them.PycURL 7.19.0 has a reference counting bug leading to a crash aftera certain number of performed requests.
Since a unittest depended on this function for a test, it is replaced...
http.client: Remove HTTP client pool code
This patch removes all remains of the HTTP client pool. Newly added unittestsprovide 96% coverage on http.client.
http.client: Show pending requests as “owner”
In the context of the lock monitor a “pending” item does not yet own therequested resource. Since these HTTP requests are already undergoingthey should be shown as owners.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
rpc/http: Show pending RPC requests in lock monitor
Not all requests use an instance of RpcRunner yet and therefore won'tshow up (only instances have access to the global Ganeti context).Currently only the IP address is accessible. Another patch will add a...
http.client: Add nice name to requests
With this change a node name instead of the IP address can be shown forpending RPC requests:Name Pendingrpc/node18.example.com/test_delay thread:Jq1/Job692/TEST_DELAY
http.client: Factorize code interacting with cURL
This simplifies HttpClientPool.ProcessRequests significantly and will behandy for showing pending RPC requests in the lock monitor.
http.client: Reduce performance impact by assertion
Call dict.values once instead of N times.
DeprecationWarning fixes for pylint
In version 0.21, pylint unified all the disable-* (and enable-*)directives to disable (resp. enable). This leads to a lot ofDeprecationWarning being emitted even if one uses the recommendedversion of pylint (0.21.1, as stated in devnotes.rst)....
PEP8 style fixes
Identified using the “pep8” utility.
Most boring patch ever
s/'/"/ in (hopefully) the right places.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
http.client: Make debug log less noisy
The HTTP client code generates quite a lot of debug log messages. Withthis patch they're hidden unless explicitely enabled in the code.
http.client: Disable SSL session ID cache
This patch disables the SSL session ID cache for all cURL operations.This is needed because http.HttpBase's PyOpenSSL implementation does notcurrently set a context using SSL_set_session_id_context(3SSL), cURLtries to re-use the session ID and, according to...
http.auth: Fix docstring error
This was missing from commit 2287b920.
Merge branch 'stable-2.2'
http.auth.ReadPasswordFile: Don't read file directly
Reading the file before this function allows for better errorreporting.
Set list of trusted SSL CAs for client to verify
As per SSL_CTX_set_client_CA_list(3SSL), set the list of acceptable CAsadvertised to SSL clients to include the server's own certificate. Thisevidently fixes the pycurl/gnutls RPC client.
During the TLS Handshake, when client verification is requested, the...
Merge branch 'devel-2.2'
Fix pylint warning in http/__init__.py
My bad for not seeing this before:R0201:614:HttpBase.GetSslCiphers: Method could be a function
Allow SSL ciphers to be overridden in HTTP server
Users of this class, such as the RAPI server, might want to override or adjustthe default SSL cipher defined in a constant.
Make family argument in FormatAddress optional
By doing this we delegate the task of finding the correct address familyto the FormatAddress method.
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
rapi.client, http.client: Format url correctly when using IPv6
This patch moves the FormatAddress helper function from daemon.py tonetutils.py. This enables its use in http.client as well as inrapi.client. Furthermore this adds functionality to format IPv6...
Support IPv6 in lib/http/server.py
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Convert RPC client to PycURL
Instead of using our custom HTTP client, using PycURL's multiinterface allows us to get rid of the HTTP client threadpool.The majority of the code is still in the ganeti.http.clientmodule.
A simple per-thread HTTP client pool gives cURL a chance to...
Disallow DES for SSL connections
Older OpenSSL versions include DES-CBC3-* ciphers when specifying theHIGH group of ciphers. Removing potentially weak ciphers from the listof allowed ciphers ensures only strong ciphers are considered for SSLconnections....
http client: support per-request read timeout
Currently, the read timeout is hardcoded in theHttpClientRequestExecutor class. The patch changes the timeout so thatit's a per-request property, and makes the rpc.Client class pass oneexplicitly in. Furthermore, we modify the rpc.RpcRunner class to support...
Move hash functions to the compat module
Since the hash functions' changed their module name between python 2.4and 2.6, and we have to do an try/import/except trick, we'll do it justonce, for both hash functions, and in compat.py. This also fixes a use...
Merge branch 'devel-2.1'
move http.WaitForSocketCondition to utils
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
WaitForSocketCondition: rename, handle EINTR
- Rename WaitForSocketCondition to SingleWaitForFdCondition - Avoid potentially infinite loop, if we continue to get interrupted - Handle eintr correctly - Avoid the poller try/finally, as the poller object gets destroyed...
Merge remote branch 'origin/devel-2.1'
http.auth: Disable pylint warnings
http.server: Improve request logging in debug mode
Provide unittests for http.auth
To simplify writing unittests, one data structure class in http.server isalso changed. According to the coverage utility, this provides 95%coverage.
http.auth: Fix bug with checking hashed passwords
When username and password were sent for a resource not requiringauthentication, it wouldn't be accepted if the user in question had ahashed password. The reason was that the function GetAuthRealm used to...
Merge remote branch 'origin/devel-2.0' into devel-2.1
Conflicts: NEWS: Trivial configure.ac: Trivial...
Fix two potentially endless loops in http library
The first can be problematic if poll(2) returns POLLHUP|POLLERR on asocket. Before it would be only be respected for SOCKOP_RECV, but sincethey can also occur on other socket operations, esp. in combination with...
Reset tempfile module after fork where useful
Merge branch 'devel-2.0' into devel-2.1
Conflicts: lib/backend.py - trivial merge...
Ensure all int/float conversions are handled right
int()/float() can raise either ValueError (in case of int("a")), orTypeError (in case of int(None)). We had many bugs over time due tothis, and a recent one was just diagnosed, so we go over the codebase...
Remove http.HttpJsonConverter
With the move of the content-type handling to the various users of the HTTPlayer, this class isn't really useful anymore.
http.server: No longer enfore JSON encoding for body
The HTTP layer shouldn't care about the contents of the request data orresponses. This requires further changes in the RAPI code to handle clientrequests correctly.
http.server: Refuse HTTP/1.1 request without Host header
http: Add two new exceptions, one constant
These will be useful in the future in case we don't enfore JSON encodinganymore in the http.server module. The HTTP 1.1 RFC recommends error 415(Unsupported Media Type) to be returned in case the client requests an...
Factorize LUXI parsing and handling code
Also fix a typo in http/__init__.py and add unittestsfor the LUXI parsing and formatting functions.
Improve logging for workerpool tasks by providing repr
Before it would log something like “starting task(<ganeti.http.client._HttpClientPendingRequest object at 0x2aaaad176790>,)”,which isn't really useful for debugging. Now it'll log “[…]<ganeti.http.client._HttpClientPendingRequest...
workerpool: Make worker ID alphanumeric
Having a proper name instead of just a number makes debuggingeasier.
Merge remote branch 'devel-2.1'
Fix unused imports or add silences where needed
In some cases pylint doesn't parse the import correctly, so we addsilences; but there are also many cases of unused imports, which wesimply remove.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Olivier Tharan <olive@google.com>
Further pylint disables, mostly for Unused args
Many of our functions have to follow a given API, and thus we have tokeep a given signature, but pylint doesn't understand this. Therefore,we silence this warning.
The patch does a few other cleanups.
Signed-off-by: Iustin Pop <iustin@google.com>...
Convert to static methods (where appropriate)
Many methods are simple pure functions, and not depending on the objectstate. We convert these to staticmethods.
Add targeted pylint disables
This patch should have only:
- pylint disables- docstring changes- whitespace changes
Remove many 'Unused variable' warnings
Note there are some cases left which need extra cleanup.
Add targetted pylint disables
This patch adds targeted pylint disables, where it makes sense (eitherdue to limitations in pylint or due to historical usage), and also a fewblanket ones in rapi where all the names are… “different”.
A few style updates
Add check for OpenSSL entropy status
By checking for this explicitly, the errors (SSLEAY_RAND_BYTES, “PRNGnot seeded”) will happen in the start-up phase of the daemon and notonly when executing remote procedure calls.
Fix pylint 'E' (error) codes
This patch adds some silences and tweaks the code slightly so that“pylint --rcfile pylintrc -e ganeti” doesn't give any errors.
The biggest change is in jqueue.py, the move of _RequireOpenQueue out ofthe JobQueue class. Since that is actually a function and not a method...
Epydoc fixes
http.auth: Add new function to verify passwords
This new function supports two schemes for passwords:- Old-style cleartext passwords- Hashed passwords according to RFC2617 (H(A1))
Schemes are differentiated by their prefix, a concept alsoused in OpenLDAP. Cleartext passwords can no longer start...
ganeti-noded: Close listening socket in child
Convert the http server/mainloop to asyncore
We can avoid most of the Mainloop.Run() code if we use asyncorefor delivering I/O events, and just concentrate on what's missing inasyncore: singnal handling and timers. This way confd can be ported touse Mainloop as well....
Fix pylint warnings
Fix some typos
Fix HTTP server library handling of credentials
Currently the http library only checks credentials when authenticationis required. This means that any credentials are accepted on the rootresource, for example, which makes problems hard to diagnose - the...
Fix _NOQUOTE regexp
Allow expressions longer than one character to match.
Reviewed-by: imsnah
Fix some epydoc style issues
99% of the epydoc return tags are "@return:", but each of the modified fileshad one "@returns:" line. We fix this for consistency.
RAPI: format error messages as JSON
This patch changes the format of the HTTP error messages from text/html, whichis hard to parse from RAPI clients, to JSON which can be automatically parsed.
The error message is an object, which contains always three keys:...
Make RAPI return 502/504 errors for luxi errors
This changes the RAPI error codes for luxi errors; a timeout error isnow reported properly as 504, while any other luxi error is reported as502.
It would be good to convert even more errors into proper return codes in...
rapi: fix authentication and queries
For queries, we don't want to require authentication. We fix this by adding anoverride GetAuthRealm in the rapi daemon.
We also fix a method name.
Some docstring updates
This patch rewraps some comments to shorter lengths, changesdouble-quotes to single-quotes inside triple-quoted docstrings forbetter editor handling.
It also fixes some epydoc errors, namely invalid crossreferences (aftermethod rename), documentation for inexistent (removed) parameters, etc....
ganeti-noded: reduce log noise
The source port/addr is currently logged three times for eachconnection, and this is unnecessary. We change two log entries to debug,since they are useful for precise timing, and we keep only one at INFOlevel.
Fix some pylint-detected issues
Two bad indentation cases and a missing variable.
ganeti.http: Function to read password file
Lines in the password file are of the following format:
<username> <password> [options]
Fields are separated by whitespace. Username and password aremandatory, options are optional and separated by comma (",")....
ganeti.http: Add support for private data in HTTP requests
Reviewed-by: amishchenko
ganeti.http: Add support for basic HTTP authentication
As per RFC2617.
ganeti.http: Prepare authentication for HTTP server
The authentication class will override PreHandleRequest.
ganeti.http: Don't pass poller object around
They're cheap to instantiate and doing this changes makes the codea bit simpler.
Reviewed-by: ultrotter
Rename http.HttpInternalError to HttpInternalServerError
All other exceptions are named after the error name in RFC2616 (HTTP/1.1).
ganeti.http: Add more constants and errors
ganeti.http: Ignore ENOTCONN when shutting down the connection
Implement support for additional headers with HTTP errors
Add simple unittests for ganeti.http
More complex unittests will need some refactoring in the HTTP code.