root / doc / design-node-security.rst @ 0565f862
History | View | Annotate | Download (20.5 kB)
1 |
============================= |
---|---|
2 |
Improvements of Node Security |
3 |
============================= |
4 |
|
5 |
This document describes an enhancement of Ganeti's security by restricting |
6 |
the distribution of security-sensitive data to the master and master |
7 |
candidates only. |
8 |
|
9 |
Note: In this document, we will use the term 'normal node' for a node that |
10 |
is neither master nor master-candidate. |
11 |
|
12 |
.. contents:: :depth: 4 |
13 |
|
14 |
Objective |
15 |
========= |
16 |
|
17 |
Up till 2.10, Ganeti distributes security-relevant keys to all nodes, |
18 |
including nodes that are neither master nor master-candidates. Those |
19 |
keys are the private and public SSH keys for node communication and the |
20 |
SSL certficate and private key for RPC communication. Objective of this |
21 |
design is to limit the set of nodes that can establish ssh and RPC |
22 |
connections to the master and master candidates. |
23 |
|
24 |
As pointed out in |
25 |
`issue 377 <https://code.google.com/p/ganeti/issues/detail?id=377>`_, this |
26 |
is a security risk. Since all nodes have these keys, compromising |
27 |
any of those nodes would possibly give an attacker access to all other |
28 |
machines in the cluster. Reducing the set of nodes that are able to |
29 |
make ssh and RPC connections to the master and master candidates would |
30 |
significantly reduce the risk simply because fewer machines would be a |
31 |
valuable target for attackers. |
32 |
|
33 |
Note: For bigger installations of Ganeti, it is advisable to run master |
34 |
candidate nodes as non-vm-capable nodes. This would reduce the attack |
35 |
surface for the hypervisor exploitation. |
36 |
|
37 |
|
38 |
Detailed design |
39 |
=============== |
40 |
|
41 |
|
42 |
Current state and shortcomings |
43 |
------------------------------ |
44 |
|
45 |
Currently (as of 2.10), all nodes hold the following information: |
46 |
|
47 |
- the ssh host keys (public and private) |
48 |
- the ssh root keys (public and private) |
49 |
- node daemon certificate (the SSL client certificate and its |
50 |
corresponding private key) |
51 |
|
52 |
Concerning ssh, this setup contains the following security issue. Since |
53 |
all nodes of a cluster can ssh as root into any other cluster node, one |
54 |
compromised node can harm all other nodes of a cluster. |
55 |
|
56 |
Regarding the SSL encryption of the RPC communication with the node |
57 |
daemon, we currently have the following setup. There is only one |
58 |
certificate which is used as both, client and server certificate. Besides |
59 |
the SSL client verification, we check if the used client certificate is |
60 |
the same as the certificate stored on the server. |
61 |
|
62 |
This means that any node running a node daemon can also act as an RPC |
63 |
client and use it to issue RPC calls to other cluster nodes. This in |
64 |
turn means that any compromised node could be used to make RPC calls to |
65 |
any node (including itself) to gain full control over VMs. This could |
66 |
be used by an attacker to for example bring down the VMs or exploit bugs |
67 |
in the virtualization stacks to gain access to the host machines as well. |
68 |
|
69 |
|
70 |
Proposal concerning SSH key distribution |
71 |
---------------------------------------- |
72 |
|
73 |
We propose two improvements regarding the ssh keys: |
74 |
|
75 |
#. Limit the distribution of the private ssh key to the master candidates. |
76 |
|
77 |
#. Use different ssh key pairs for each master candidate. |
78 |
|
79 |
We propose to limit the set of nodes holding the private root user SSH key |
80 |
to the master and the master candidates. This way, the security risk would |
81 |
be limited to a rather small set of nodes even though the cluster could |
82 |
consists of a lot more nodes. The set of master candidates could be protected |
83 |
better than the normal nodes (for example residing in a DMZ) to enhance |
84 |
security even more if the administrator wishes so. The following |
85 |
sections describe in detail which Ganeti commands are affected by this |
86 |
change and in what way. |
87 |
|
88 |
Security will be even more increased if each master candidate gets |
89 |
its own ssh private/public key pair. This way, one can remove a |
90 |
compromised master candidate from a cluster (including removing it's |
91 |
public key from all nodes' ``authorized_keys`` file) without having to |
92 |
regenerate and distribute new ssh keys for all master candidates. (Even |
93 |
though it is be good practice to do that anyway, since the compromising |
94 |
of the other master candidates might have taken place already.) However, |
95 |
this improvement was not part of the original feature request and |
96 |
increases the complexity of node management even more. We therefore |
97 |
consider it as second step in this design and will address |
98 |
this after the other parts of this design are implemented. |
99 |
|
100 |
The following sections describe in detail which Ganeti commands are affected |
101 |
by the first part of ssh-related improvements, limiting the key |
102 |
distribution to master candidates only. |
103 |
|
104 |
|
105 |
(Re-)Adding nodes to a cluster |
106 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
107 |
|
108 |
According to :doc:`design-node-add`, Ganeti transfers the ssh keys to |
109 |
every node that gets added to the cluster. |
110 |
|
111 |
We propose to change this procedure to treat master candidates and normal |
112 |
nodes differently. For master candidates, the procedure would stay as is. |
113 |
For normal nodes, Ganeti would transfer the public and private ssh host |
114 |
keys (as before) and only the public root key. |
115 |
|
116 |
A normal node would not be able to connect via ssh to other nodes, but |
117 |
the master (and potentially master candidates) can connect to this node. |
118 |
|
119 |
In case of readding a node that used to be in the cluster before, |
120 |
handling of the ssh keys would basically be the same with the following |
121 |
additional modifications: if the node used to be a master or |
122 |
master-candidate node, but will be a normal node after readding, Ganeti |
123 |
should make sure that the private root key is deleted if it is still |
124 |
present on the node. |
125 |
|
126 |
|
127 |
Pro- and demoting a node to/from master candidate |
128 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
129 |
|
130 |
If the role of a node is changed from 'normal' to 'master_candidate', the |
131 |
master node should at that point copy the private root ssh key. When demoting |
132 |
a node from master candidate to a normal node, the key that have been copied |
133 |
there on promotion or addition should be removed again. |
134 |
|
135 |
This affected the behavior of the following commands: |
136 |
|
137 |
:: |
138 |
gnt-node modify --master-candidate=yes |
139 |
gnt-node modify --master-candidate=no [--auto-promote] |
140 |
|
141 |
If the node has been master candidate already before the command to promote |
142 |
it was issued, Ganeti does not do anything. |
143 |
|
144 |
Note that when you demote a node from master candidate to normal node, another |
145 |
master-capable and normal node will be promoted to master candidate. For this |
146 |
newly promoted node, the same changes apply as if it was explicitely promoted. |
147 |
|
148 |
The same behavior should be ensured for the corresponding rapi command. |
149 |
|
150 |
|
151 |
Offlining and onlining a node |
152 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
153 |
|
154 |
When offlining a node, it immediately loses its role as master or master |
155 |
candidate as well. When it is onlined again, it will become master |
156 |
candidate again if it was so before. The handling of the keys should be done |
157 |
in the same way as when the node is explicitely promoted or demoted to or from |
158 |
master candidate. See the previous section for details. |
159 |
|
160 |
This affects the command: |
161 |
|
162 |
:: |
163 |
gnt-node modify --offline=yes |
164 |
gnt-node modify --offline=no [--auto-promote] |
165 |
|
166 |
For offlining, the removal of the keys is particularly important, as the |
167 |
detection of a compromised node might be the very reason for the offlining. |
168 |
Of course we cannot guarantee that removal of the key is always successful, |
169 |
because the node might not be reachable anymore. Even though it is a |
170 |
best-effort operation, it is still an improvement over the status quo, |
171 |
because currently Ganeti does not even try to remove any keys. |
172 |
|
173 |
The same behavior should be ensured for the corresponding rapi command. |
174 |
|
175 |
|
176 |
Cluster verify |
177 |
~~~~~~~~~~~~~~ |
178 |
|
179 |
To make sure the private root ssh key was not distributed to a normal |
180 |
node, 'gnt-cluster verify' will be extended by a check for the key |
181 |
on normal nodes. Additionally, it will check if the private key is |
182 |
indeed present on master candidates. |
183 |
|
184 |
|
185 |
|
186 |
Proposal regarding node daemon certificates |
187 |
------------------------------------------- |
188 |
|
189 |
Regarding the node daemon certificates, we propose the following changes |
190 |
in the design. |
191 |
|
192 |
- Instead of using the same certificate for all nodes as both, server |
193 |
and client certificate, we generate a common server certificate (and |
194 |
the corresponding private key) for all nodes and a different client |
195 |
certificate (and the corresponding private key) for each node. All |
196 |
those certificates will be self-signed for now. The client |
197 |
certificates will use the node UUID as serial number to ensure |
198 |
uniqueness within the cluster. |
199 |
- In addition, we store a mapping of |
200 |
(node UUID, client certificate digest) in the cluster's configuration |
201 |
and ssconf for hosts that are master or master candidate. |
202 |
The client certificate digest is a hash of the client certificate. |
203 |
We suggest a 'sha1' hash here. We will call this mapping 'candidate map' |
204 |
from here on. |
205 |
- The node daemon will be modified in a way that on an incoming RPC |
206 |
request, it first performs a client verification (same as before) to |
207 |
ensure that the requesting host is indeed the holder of the |
208 |
corresponding private key. Additionally, it compares the digest of |
209 |
the certificate of the incoming request to the respective entry of |
210 |
the candidate map. If the digest does not match the entry of the host |
211 |
in the mapping or is not included in the mapping at all, the SSL |
212 |
connection is refused. |
213 |
|
214 |
This design has the following advantages: |
215 |
|
216 |
- A compromised normal node cannot issue RPC calls, because it will |
217 |
not be in the candidate map. (See the ``Drawbacks`` section regarding |
218 |
an indirect way of achieving this though.) |
219 |
- A compromised master candidate would be able to issue RPC requests, |
220 |
but on detection of its compromised state, it can be removed from the |
221 |
cluster (and thus from the candidate map) without the need for |
222 |
redistribution of any certificates, because the other master candidates |
223 |
can continue using their own certificates. However, it is best |
224 |
practice to issue a complete key renewal even in this case, unless one |
225 |
can ensure no actions compromising other nodes have not already been |
226 |
carried out. |
227 |
- A compromised node would not be able to use the other (possibly master |
228 |
candidate) nodes' information from the candidate map to issue RPCs, |
229 |
because the config just stores the digests and not the certificate |
230 |
itself. |
231 |
- A compromised node would be able to obtain another node's certificate |
232 |
by waiting for incoming RPCs from this other node. However, the node |
233 |
cannot use the certificate to issue RPC calls, because the SSL client |
234 |
verification would require the node to hold the corresponding private |
235 |
key as well. |
236 |
|
237 |
Drawbacks of this design: |
238 |
|
239 |
- Complexity of node and certificate management will be increased (see |
240 |
following sections for details). |
241 |
- If the candidate map is not distributed fast enough to all nodes after |
242 |
an update of the configuration, it might be possible to issue RPC calls |
243 |
from a compromised master candidate node that has been removed |
244 |
from the Ganeti cluster already. However, this is still a better |
245 |
situation than before and an inherent problem when one wants to |
246 |
distinguish between master candidates and normal nodes. |
247 |
- A compromised master candidate would still be able to issue RPC calls, |
248 |
if it uses ssh to retrieve another master candidate's client |
249 |
certificate and the corresponding private SSL key. This is an issue |
250 |
even with the first part of the improved handling of ssh keys in this |
251 |
design (limiting ssh keys to master candidates), but it will be |
252 |
eliminated with the second part of the design (separate ssh keys for |
253 |
each master candidate). |
254 |
- Even though this proposal is an improvement towards the previous |
255 |
situation in Ganeti, it still does not use the full power of SSL. For |
256 |
further improvements, see Section "Related and future work". |
257 |
|
258 |
Alternative proposals: |
259 |
|
260 |
- Instead of generating a client certificate per node, one could think |
261 |
of just generating two different client certificates, one for normal |
262 |
nodes and one for master candidates. Noded could then just check if |
263 |
the requesting node has the master candidate certificate. Drawback of |
264 |
this proposal is that once one master candidate gets compromised, all |
265 |
master candidates would need to get a new certificate even if the |
266 |
compromised master candidate had not yet fetched the certificates |
267 |
from the other master candidates via ssh. |
268 |
- In addition to our main proposal, one could think of including a |
269 |
piece of data (for example the node's host name or UUID) in the RPC |
270 |
call which is encrypted with the requesting node's private key. The |
271 |
node daemon could check if the datum can be decrypted using the node's |
272 |
certificate. However, this would ensure similar functionality as |
273 |
SSL's built-in client verification and add significant complexity |
274 |
to Ganeti's RPC protocol. |
275 |
|
276 |
In the following sections, we describe how our design affects various |
277 |
Ganeti operations. |
278 |
|
279 |
|
280 |
Cluster initialization |
281 |
~~~~~~~~~~~~~~~~~~~~~~ |
282 |
|
283 |
On cluster initialization, so far only the node daemon certificate was |
284 |
created. With our design, two certificates (and corresponding keys) |
285 |
need to be created, a server certificate to be distributed to all nodes |
286 |
and a client certificate only to be used by this particular node. In the |
287 |
following, we use the term node daemon certificate for the server |
288 |
certficate only. |
289 |
|
290 |
In the cluster configuration, the candidate map is created. It is |
291 |
populated with the respective entry for the master node. It is also |
292 |
written to ssconf. |
293 |
|
294 |
|
295 |
(Re-)Adding nodes |
296 |
~~~~~~~~~~~~~~~~~ |
297 |
|
298 |
When a node is added, the server certificate is copied to the node (as |
299 |
before). Additionally, a new client certificate (and the corresponding |
300 |
private key) is created on the new node to be used only by the new node |
301 |
as client certifcate. |
302 |
|
303 |
If the new node is a master candidate, the candidate map is extended by |
304 |
the new node's data. As before, the updated configuration is distributed |
305 |
to all nodes (as complete configuration on the master candidates and |
306 |
ssconf on all nodes). Note that distribution of the configuration after |
307 |
adding a node is already implemented, since all nodes hold the list of |
308 |
nodes in the cluster in ssconf anyway. |
309 |
|
310 |
If the configuration for whatever reason already holds an entry for this |
311 |
node, it will be overriden. |
312 |
|
313 |
When readding a node, the procedure is the same as for adding a node. |
314 |
|
315 |
|
316 |
Promotion and demotion of master candidates |
317 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
318 |
|
319 |
When a normal node gets promoted to be master candidate, an entry to the |
320 |
candidate map has to be added and the updated configuration has to be |
321 |
distributed to all nodes. If there was already an entry for the node, |
322 |
we override it. |
323 |
|
324 |
On demotion of a master candidate, the node's entry in the candidate map |
325 |
gets removed and the updated configuration gets redistibuted. |
326 |
|
327 |
The same procedure applied to onlining and offlining master candidates. |
328 |
|
329 |
|
330 |
Cluster verify |
331 |
~~~~~~~~~~~~~~ |
332 |
|
333 |
Cluster verify will be extended by the following checks: |
334 |
|
335 |
- Whether each entry in the candidate map indeed corresponds to a master |
336 |
candidate. |
337 |
- Whether the master candidate's certificate digest match their entry |
338 |
in the candidate map. |
339 |
- Whether no node tries to use the certificate of another node. In |
340 |
particular, it is important to check that no normal node tries to |
341 |
use the certificate of a master candidate. |
342 |
|
343 |
|
344 |
Crypto renewal |
345 |
~~~~~~~~~~~~~~ |
346 |
|
347 |
Currently, when the cluster's cryptographic tokens are renewed using the |
348 |
``gnt-cluster renew-crypto`` command the node daemon certificate is |
349 |
renewed (among others). Option ``--new-cluster-certificate`` renews the |
350 |
node daemon certificate only. |
351 |
|
352 |
By adding an option ``--new-node-certificates`` we offer to renew the |
353 |
client certificate. Whenever the client certificates are renewed, the |
354 |
candidate map has to be updated and redistributed. |
355 |
|
356 |
If for whatever reason, the candidate map becomes inconsistent, for example |
357 |
due inconsistent updating after a demotion or offlining), the user can use |
358 |
this option to renew the client certificates and update the candidate |
359 |
certificate map. |
360 |
|
361 |
|
362 |
Further considerations |
363 |
---------------------- |
364 |
|
365 |
Watcher |
366 |
~~~~~~~ |
367 |
|
368 |
The watcher is a script that is run on all nodes in regular intervals. The |
369 |
changes proposed in this design will not affect the watcher's implementation, |
370 |
because it behaves differently on the master than on non-master nodes. |
371 |
|
372 |
Only on the master, it issues query calls which would require a client |
373 |
certificate of a node in the candidate mapping. This is the case for the |
374 |
master node. On non-master node, it's only external communication is done via |
375 |
the ConfD protocol, which uses the hmac key, which is present on all nodes. |
376 |
Besides that, the watcher does not make any ssh connections, and thus is |
377 |
not affected by the changes in ssh key handling either. |
378 |
|
379 |
|
380 |
Other Keys and Daemons |
381 |
~~~~~~~~~~~~~~~~~~~~~~ |
382 |
|
383 |
Ganeti handles a couple of other keys/certificates that have not been mentioned |
384 |
in this design so far. Also, other daemons than the ones mentioned so far |
385 |
perform intra-cluster communication. Neither the keys nor the daemons will |
386 |
be affected by this design for several reasons: |
387 |
|
388 |
- The hmac key used by ConfD (see :doc:`design-2.1`): the hmac key is still |
389 |
distributed to all nodes, because it was designed to be used for |
390 |
communicating with ConfD, which should be possible from all nodes. |
391 |
For example, the monitoring daemon which runs on all nodes uses it to |
392 |
retrieve information from ConfD. However, since communication with ConfD |
393 |
is read-only, a compromised node holding the hmac key does not enable an |
394 |
attacker to change the cluster's state. |
395 |
|
396 |
- The WConfD daemon writes the configuration to all master candidates |
397 |
via RPC. Since it only runs on the master node, it's ability to run |
398 |
RPC requests is maintained with this design. |
399 |
|
400 |
- The rapi SSL key certificate and rapi user/password file 'rapi_users' is |
401 |
already only copied to the master candidates (see :doc:`design-2.1`, |
402 |
Section ``Redistribute Config``). |
403 |
|
404 |
- The spice certificates are still distributed to all nodes, since it should |
405 |
be possible to use spice to access VMs on any cluster node. |
406 |
|
407 |
- The cluster domain secret is used for inter-cluster instance moves. |
408 |
Since instances can be moved from any normal node of the source cluster to |
409 |
any normal node of the destination cluster, the presence of this |
410 |
secret on all nodes is necessary. |
411 |
|
412 |
|
413 |
Related and Future Work |
414 |
~~~~~~~~~~~~~~~~~~~~~~~ |
415 |
|
416 |
There a couple of suggestions on how to improve the SSL setup even more. |
417 |
As a trade-off wrt to complexity and implementation effort, we did not |
418 |
implement them yet (as of version 2.11) but describe them here for |
419 |
future reference. |
420 |
|
421 |
- All SSL certificates that Ganeti uses so far are self-signed. It would |
422 |
increase the security if they were signed by a common CA. There is |
423 |
already a design doc for a Ganeti CA which was suggested in a |
424 |
different context (related to import/export). This would also be a |
425 |
benefit for the RPC calls. See design doc :doc:`design-impexp2` for |
426 |
more information. Implementing a CA is rather complex, because it |
427 |
would mean also to support renewing the CA certificate and providing |
428 |
and supporting infrastructure to revoke compromised certificates. |
429 |
- An extension of the previous suggestion would be to even enable the |
430 |
system administrator to use an external CA. Especially in bigger |
431 |
setups, where already an SSL infrastructure exists, it would be useful |
432 |
if Ganeti can simply be integrated with it, rather than forcing the |
433 |
user to use the Ganeti CA. |
434 |
- A lighter version of using a CA would be to use the server certificate |
435 |
to sign the client certificate instead of using self-signed |
436 |
certificates for both. The probleme here is that this would make |
437 |
renewing the server certificate rather complicated, because all client |
438 |
certificates would need to be resigned and redistributed as well, |
439 |
which leads to interesting chicken-and-egg problems when this is done |
440 |
via RPC calls. |
441 |
- Ganeti RPC calls are currently done without checking if the hostname |
442 |
of the node complies with the common name of the certificate. This |
443 |
might be a desirable feature, but would increase the effort when a |
444 |
node is renamed. |
445 |
- The typical use case for SSL is to have one certificate per node |
446 |
rather than one shared certificate (Ganeti's noded server certificate) |
447 |
and a client certificate. One could change the design in a way that |
448 |
only one certificate per node is used, but this would require a common |
449 |
CA so that the validity of the certificate can be established by every |
450 |
node in the cluster. |
451 |
- With the proposed design, the serial numbers of the client |
452 |
certificates are set to the node UUIDs. This is technically also not |
453 |
complying to how SSL is supposed to be used, as the serial numbers |
454 |
should reflect the enumeration of certificates created by the CA. Once |
455 |
a CA is implemented, it might be reasonable to change this |
456 |
accordingly. The implementation of the proposed design also has the |
457 |
drawback of the serial number not changing even if the certificate is |
458 |
replaced by a new one (for example when calling ``gnt-cluster renew- |
459 |
crypt``), which also does not comply to way SSL was designed to be |
460 |
used. |
461 |
|
462 |
.. vim: set textwidth=72 : |
463 |
.. Local Variables: |
464 |
.. mode: rst |
465 |
.. fill-column: 72 |
466 |
.. End: |