root / doc / design-node-security.rst @ 0565f862
History | View | Annotate | Download (20.5 kB)
1 | c2a3c07d | Helga Velroyen | ============================= |
---|---|---|---|
2 | c2a3c07d | Helga Velroyen | Improvements of Node Security |
3 | c2a3c07d | Helga Velroyen | ============================= |
4 | c2a3c07d | Helga Velroyen | |
5 | c2a3c07d | Helga Velroyen | This document describes an enhancement of Ganeti's security by restricting |
6 | c2a3c07d | Helga Velroyen | the distribution of security-sensitive data to the master and master |
7 | c2a3c07d | Helga Velroyen | candidates only. |
8 | c2a3c07d | Helga Velroyen | |
9 | c2a3c07d | Helga Velroyen | Note: In this document, we will use the term 'normal node' for a node that |
10 | c2a3c07d | Helga Velroyen | is neither master nor master-candidate. |
11 | c2a3c07d | Helga Velroyen | |
12 | c2a3c07d | Helga Velroyen | .. contents:: :depth: 4 |
13 | c2a3c07d | Helga Velroyen | |
14 | c2a3c07d | Helga Velroyen | Objective |
15 | c2a3c07d | Helga Velroyen | ========= |
16 | c2a3c07d | Helga Velroyen | |
17 | c2a3c07d | Helga Velroyen | Up till 2.10, Ganeti distributes security-relevant keys to all nodes, |
18 | c2a3c07d | Helga Velroyen | including nodes that are neither master nor master-candidates. Those |
19 | c2a3c07d | Helga Velroyen | keys are the private and public SSH keys for node communication and the |
20 | c2a3c07d | Helga Velroyen | SSL certficate and private key for RPC communication. Objective of this |
21 | c2a3c07d | Helga Velroyen | design is to limit the set of nodes that can establish ssh and RPC |
22 | c2a3c07d | Helga Velroyen | connections to the master and master candidates. |
23 | c2a3c07d | Helga Velroyen | |
24 | c2a3c07d | Helga Velroyen | As pointed out in |
25 | c2a3c07d | Helga Velroyen | `issue 377 <https://code.google.com/p/ganeti/issues/detail?id=377>`_, this |
26 | c2a3c07d | Helga Velroyen | is a security risk. Since all nodes have these keys, compromising |
27 | c2a3c07d | Helga Velroyen | any of those nodes would possibly give an attacker access to all other |
28 | c2a3c07d | Helga Velroyen | machines in the cluster. Reducing the set of nodes that are able to |
29 | c2a3c07d | Helga Velroyen | make ssh and RPC connections to the master and master candidates would |
30 | c2a3c07d | Helga Velroyen | significantly reduce the risk simply because fewer machines would be a |
31 | c2a3c07d | Helga Velroyen | valuable target for attackers. |
32 | c2a3c07d | Helga Velroyen | |
33 | c2a3c07d | Helga Velroyen | Note: For bigger installations of Ganeti, it is advisable to run master |
34 | c2a3c07d | Helga Velroyen | candidate nodes as non-vm-capable nodes. This would reduce the attack |
35 | c2a3c07d | Helga Velroyen | surface for the hypervisor exploitation. |
36 | c2a3c07d | Helga Velroyen | |
37 | c2a3c07d | Helga Velroyen | |
38 | c2a3c07d | Helga Velroyen | Detailed design |
39 | c2a3c07d | Helga Velroyen | =============== |
40 | c2a3c07d | Helga Velroyen | |
41 | c2a3c07d | Helga Velroyen | |
42 | c2a3c07d | Helga Velroyen | Current state and shortcomings |
43 | c2a3c07d | Helga Velroyen | ------------------------------ |
44 | c2a3c07d | Helga Velroyen | |
45 | c2a3c07d | Helga Velroyen | Currently (as of 2.10), all nodes hold the following information: |
46 | c2a3c07d | Helga Velroyen | |
47 | c2a3c07d | Helga Velroyen | - the ssh host keys (public and private) |
48 | c2a3c07d | Helga Velroyen | - the ssh root keys (public and private) |
49 | 575b31bf | Helga Velroyen | - node daemon certificate (the SSL client certificate and its |
50 | c2a3c07d | Helga Velroyen | corresponding private key) |
51 | c2a3c07d | Helga Velroyen | |
52 | c2a3c07d | Helga Velroyen | Concerning ssh, this setup contains the following security issue. Since |
53 | c2a3c07d | Helga Velroyen | all nodes of a cluster can ssh as root into any other cluster node, one |
54 | c2a3c07d | Helga Velroyen | compromised node can harm all other nodes of a cluster. |
55 | c2a3c07d | Helga Velroyen | |
56 | c2a3c07d | Helga Velroyen | Regarding the SSL encryption of the RPC communication with the node |
57 | c2a3c07d | Helga Velroyen | daemon, we currently have the following setup. There is only one |
58 | c2a3c07d | Helga Velroyen | certificate which is used as both, client and server certificate. Besides |
59 | c2a3c07d | Helga Velroyen | the SSL client verification, we check if the used client certificate is |
60 | c2a3c07d | Helga Velroyen | the same as the certificate stored on the server. |
61 | c2a3c07d | Helga Velroyen | |
62 | c2a3c07d | Helga Velroyen | This means that any node running a node daemon can also act as an RPC |
63 | c2a3c07d | Helga Velroyen | client and use it to issue RPC calls to other cluster nodes. This in |
64 | c2a3c07d | Helga Velroyen | turn means that any compromised node could be used to make RPC calls to |
65 | c2a3c07d | Helga Velroyen | any node (including itself) to gain full control over VMs. This could |
66 | c2a3c07d | Helga Velroyen | be used by an attacker to for example bring down the VMs or exploit bugs |
67 | c2a3c07d | Helga Velroyen | in the virtualization stacks to gain access to the host machines as well. |
68 | c2a3c07d | Helga Velroyen | |
69 | c2a3c07d | Helga Velroyen | |
70 | c2a3c07d | Helga Velroyen | Proposal concerning SSH key distribution |
71 | c2a3c07d | Helga Velroyen | ---------------------------------------- |
72 | c2a3c07d | Helga Velroyen | |
73 | c2a3c07d | Helga Velroyen | We propose two improvements regarding the ssh keys: |
74 | c2a3c07d | Helga Velroyen | |
75 | c2a3c07d | Helga Velroyen | #. Limit the distribution of the private ssh key to the master candidates. |
76 | c2a3c07d | Helga Velroyen | |
77 | c2a3c07d | Helga Velroyen | #. Use different ssh key pairs for each master candidate. |
78 | c2a3c07d | Helga Velroyen | |
79 | c2a3c07d | Helga Velroyen | We propose to limit the set of nodes holding the private root user SSH key |
80 | c2a3c07d | Helga Velroyen | to the master and the master candidates. This way, the security risk would |
81 | c2a3c07d | Helga Velroyen | be limited to a rather small set of nodes even though the cluster could |
82 | c2a3c07d | Helga Velroyen | consists of a lot more nodes. The set of master candidates could be protected |
83 | c2a3c07d | Helga Velroyen | better than the normal nodes (for example residing in a DMZ) to enhance |
84 | c2a3c07d | Helga Velroyen | security even more if the administrator wishes so. The following |
85 | c2a3c07d | Helga Velroyen | sections describe in detail which Ganeti commands are affected by this |
86 | c2a3c07d | Helga Velroyen | change and in what way. |
87 | c2a3c07d | Helga Velroyen | |
88 | c2a3c07d | Helga Velroyen | Security will be even more increased if each master candidate gets |
89 | c2a3c07d | Helga Velroyen | its own ssh private/public key pair. This way, one can remove a |
90 | c2a3c07d | Helga Velroyen | compromised master candidate from a cluster (including removing it's |
91 | c2a3c07d | Helga Velroyen | public key from all nodes' ``authorized_keys`` file) without having to |
92 | c2a3c07d | Helga Velroyen | regenerate and distribute new ssh keys for all master candidates. (Even |
93 | c2a3c07d | Helga Velroyen | though it is be good practice to do that anyway, since the compromising |
94 | c2a3c07d | Helga Velroyen | of the other master candidates might have taken place already.) However, |
95 | c2a3c07d | Helga Velroyen | this improvement was not part of the original feature request and |
96 | c2a3c07d | Helga Velroyen | increases the complexity of node management even more. We therefore |
97 | c2a3c07d | Helga Velroyen | consider it as second step in this design and will address |
98 | c2a3c07d | Helga Velroyen | this after the other parts of this design are implemented. |
99 | c2a3c07d | Helga Velroyen | |
100 | c2a3c07d | Helga Velroyen | The following sections describe in detail which Ganeti commands are affected |
101 | c2a3c07d | Helga Velroyen | by the first part of ssh-related improvements, limiting the key |
102 | c2a3c07d | Helga Velroyen | distribution to master candidates only. |
103 | c2a3c07d | Helga Velroyen | |
104 | c2a3c07d | Helga Velroyen | |
105 | c2a3c07d | Helga Velroyen | (Re-)Adding nodes to a cluster |
106 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
107 | c2a3c07d | Helga Velroyen | |
108 | 0565f862 | Helga Velroyen | According to :doc:`design-node-add`, Ganeti transfers the ssh keys to |
109 | 0565f862 | Helga Velroyen | every node that gets added to the cluster. |
110 | c2a3c07d | Helga Velroyen | |
111 | c2a3c07d | Helga Velroyen | We propose to change this procedure to treat master candidates and normal |
112 | c2a3c07d | Helga Velroyen | nodes differently. For master candidates, the procedure would stay as is. |
113 | c2a3c07d | Helga Velroyen | For normal nodes, Ganeti would transfer the public and private ssh host |
114 | c2a3c07d | Helga Velroyen | keys (as before) and only the public root key. |
115 | c2a3c07d | Helga Velroyen | |
116 | c2a3c07d | Helga Velroyen | A normal node would not be able to connect via ssh to other nodes, but |
117 | c2a3c07d | Helga Velroyen | the master (and potentially master candidates) can connect to this node. |
118 | c2a3c07d | Helga Velroyen | |
119 | c2a3c07d | Helga Velroyen | In case of readding a node that used to be in the cluster before, |
120 | c2a3c07d | Helga Velroyen | handling of the ssh keys would basically be the same with the following |
121 | c2a3c07d | Helga Velroyen | additional modifications: if the node used to be a master or |
122 | c2a3c07d | Helga Velroyen | master-candidate node, but will be a normal node after readding, Ganeti |
123 | c2a3c07d | Helga Velroyen | should make sure that the private root key is deleted if it is still |
124 | c2a3c07d | Helga Velroyen | present on the node. |
125 | c2a3c07d | Helga Velroyen | |
126 | c2a3c07d | Helga Velroyen | |
127 | c2a3c07d | Helga Velroyen | Pro- and demoting a node to/from master candidate |
128 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
129 | c2a3c07d | Helga Velroyen | |
130 | c2a3c07d | Helga Velroyen | If the role of a node is changed from 'normal' to 'master_candidate', the |
131 | c2a3c07d | Helga Velroyen | master node should at that point copy the private root ssh key. When demoting |
132 | c2a3c07d | Helga Velroyen | a node from master candidate to a normal node, the key that have been copied |
133 | c2a3c07d | Helga Velroyen | there on promotion or addition should be removed again. |
134 | c2a3c07d | Helga Velroyen | |
135 | c2a3c07d | Helga Velroyen | This affected the behavior of the following commands: |
136 | c2a3c07d | Helga Velroyen | |
137 | c2a3c07d | Helga Velroyen | :: |
138 | c2a3c07d | Helga Velroyen | gnt-node modify --master-candidate=yes |
139 | c2a3c07d | Helga Velroyen | gnt-node modify --master-candidate=no [--auto-promote] |
140 | c2a3c07d | Helga Velroyen | |
141 | c2a3c07d | Helga Velroyen | If the node has been master candidate already before the command to promote |
142 | c2a3c07d | Helga Velroyen | it was issued, Ganeti does not do anything. |
143 | c2a3c07d | Helga Velroyen | |
144 | c2a3c07d | Helga Velroyen | Note that when you demote a node from master candidate to normal node, another |
145 | c2a3c07d | Helga Velroyen | master-capable and normal node will be promoted to master candidate. For this |
146 | c2a3c07d | Helga Velroyen | newly promoted node, the same changes apply as if it was explicitely promoted. |
147 | c2a3c07d | Helga Velroyen | |
148 | c2a3c07d | Helga Velroyen | The same behavior should be ensured for the corresponding rapi command. |
149 | c2a3c07d | Helga Velroyen | |
150 | c2a3c07d | Helga Velroyen | |
151 | c2a3c07d | Helga Velroyen | Offlining and onlining a node |
152 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
153 | c2a3c07d | Helga Velroyen | |
154 | c2a3c07d | Helga Velroyen | When offlining a node, it immediately loses its role as master or master |
155 | c2a3c07d | Helga Velroyen | candidate as well. When it is onlined again, it will become master |
156 | c2a3c07d | Helga Velroyen | candidate again if it was so before. The handling of the keys should be done |
157 | c2a3c07d | Helga Velroyen | in the same way as when the node is explicitely promoted or demoted to or from |
158 | c2a3c07d | Helga Velroyen | master candidate. See the previous section for details. |
159 | c2a3c07d | Helga Velroyen | |
160 | c2a3c07d | Helga Velroyen | This affects the command: |
161 | c2a3c07d | Helga Velroyen | |
162 | c2a3c07d | Helga Velroyen | :: |
163 | c2a3c07d | Helga Velroyen | gnt-node modify --offline=yes |
164 | c2a3c07d | Helga Velroyen | gnt-node modify --offline=no [--auto-promote] |
165 | c2a3c07d | Helga Velroyen | |
166 | c2a3c07d | Helga Velroyen | For offlining, the removal of the keys is particularly important, as the |
167 | c2a3c07d | Helga Velroyen | detection of a compromised node might be the very reason for the offlining. |
168 | c2a3c07d | Helga Velroyen | Of course we cannot guarantee that removal of the key is always successful, |
169 | c2a3c07d | Helga Velroyen | because the node might not be reachable anymore. Even though it is a |
170 | c2a3c07d | Helga Velroyen | best-effort operation, it is still an improvement over the status quo, |
171 | c2a3c07d | Helga Velroyen | because currently Ganeti does not even try to remove any keys. |
172 | c2a3c07d | Helga Velroyen | |
173 | c2a3c07d | Helga Velroyen | The same behavior should be ensured for the corresponding rapi command. |
174 | c2a3c07d | Helga Velroyen | |
175 | c2a3c07d | Helga Velroyen | |
176 | c2a3c07d | Helga Velroyen | Cluster verify |
177 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~ |
178 | c2a3c07d | Helga Velroyen | |
179 | c2a3c07d | Helga Velroyen | To make sure the private root ssh key was not distributed to a normal |
180 | c2a3c07d | Helga Velroyen | node, 'gnt-cluster verify' will be extended by a check for the key |
181 | c2a3c07d | Helga Velroyen | on normal nodes. Additionally, it will check if the private key is |
182 | c2a3c07d | Helga Velroyen | indeed present on master candidates. |
183 | c2a3c07d | Helga Velroyen | |
184 | c2a3c07d | Helga Velroyen | |
185 | c2a3c07d | Helga Velroyen | |
186 | c2a3c07d | Helga Velroyen | Proposal regarding node daemon certificates |
187 | c2a3c07d | Helga Velroyen | ------------------------------------------- |
188 | c2a3c07d | Helga Velroyen | |
189 | c2a3c07d | Helga Velroyen | Regarding the node daemon certificates, we propose the following changes |
190 | c2a3c07d | Helga Velroyen | in the design. |
191 | c2a3c07d | Helga Velroyen | |
192 | c2a3c07d | Helga Velroyen | - Instead of using the same certificate for all nodes as both, server |
193 | c2a3c07d | Helga Velroyen | and client certificate, we generate a common server certificate (and |
194 | c2a3c07d | Helga Velroyen | the corresponding private key) for all nodes and a different client |
195 | 0565f862 | Helga Velroyen | certificate (and the corresponding private key) for each node. All |
196 | 0565f862 | Helga Velroyen | those certificates will be self-signed for now. The client |
197 | 0565f862 | Helga Velroyen | certificates will use the node UUID as serial number to ensure |
198 | 0565f862 | Helga Velroyen | uniqueness within the cluster. |
199 | c2a3c07d | Helga Velroyen | - In addition, we store a mapping of |
200 | c2a3c07d | Helga Velroyen | (node UUID, client certificate digest) in the cluster's configuration |
201 | c2a3c07d | Helga Velroyen | and ssconf for hosts that are master or master candidate. |
202 | c2a3c07d | Helga Velroyen | The client certificate digest is a hash of the client certificate. |
203 | c2a3c07d | Helga Velroyen | We suggest a 'sha1' hash here. We will call this mapping 'candidate map' |
204 | c2a3c07d | Helga Velroyen | from here on. |
205 | c2a3c07d | Helga Velroyen | - The node daemon will be modified in a way that on an incoming RPC |
206 | c2a3c07d | Helga Velroyen | request, it first performs a client verification (same as before) to |
207 | c2a3c07d | Helga Velroyen | ensure that the requesting host is indeed the holder of the |
208 | c2a3c07d | Helga Velroyen | corresponding private key. Additionally, it compares the digest of |
209 | c2a3c07d | Helga Velroyen | the certificate of the incoming request to the respective entry of |
210 | c2a3c07d | Helga Velroyen | the candidate map. If the digest does not match the entry of the host |
211 | c2a3c07d | Helga Velroyen | in the mapping or is not included in the mapping at all, the SSL |
212 | c2a3c07d | Helga Velroyen | connection is refused. |
213 | c2a3c07d | Helga Velroyen | |
214 | c2a3c07d | Helga Velroyen | This design has the following advantages: |
215 | c2a3c07d | Helga Velroyen | |
216 | c2a3c07d | Helga Velroyen | - A compromised normal node cannot issue RPC calls, because it will |
217 | c2a3c07d | Helga Velroyen | not be in the candidate map. (See the ``Drawbacks`` section regarding |
218 | c2a3c07d | Helga Velroyen | an indirect way of achieving this though.) |
219 | c2a3c07d | Helga Velroyen | - A compromised master candidate would be able to issue RPC requests, |
220 | c2a3c07d | Helga Velroyen | but on detection of its compromised state, it can be removed from the |
221 | c2a3c07d | Helga Velroyen | cluster (and thus from the candidate map) without the need for |
222 | c2a3c07d | Helga Velroyen | redistribution of any certificates, because the other master candidates |
223 | c2a3c07d | Helga Velroyen | can continue using their own certificates. However, it is best |
224 | c2a3c07d | Helga Velroyen | practice to issue a complete key renewal even in this case, unless one |
225 | c2a3c07d | Helga Velroyen | can ensure no actions compromising other nodes have not already been |
226 | c2a3c07d | Helga Velroyen | carried out. |
227 | c2a3c07d | Helga Velroyen | - A compromised node would not be able to use the other (possibly master |
228 | c2a3c07d | Helga Velroyen | candidate) nodes' information from the candidate map to issue RPCs, |
229 | c2a3c07d | Helga Velroyen | because the config just stores the digests and not the certificate |
230 | c2a3c07d | Helga Velroyen | itself. |
231 | c2a3c07d | Helga Velroyen | - A compromised node would be able to obtain another node's certificate |
232 | c2a3c07d | Helga Velroyen | by waiting for incoming RPCs from this other node. However, the node |
233 | c2a3c07d | Helga Velroyen | cannot use the certificate to issue RPC calls, because the SSL client |
234 | c2a3c07d | Helga Velroyen | verification would require the node to hold the corresponding private |
235 | c2a3c07d | Helga Velroyen | key as well. |
236 | c2a3c07d | Helga Velroyen | |
237 | c2a3c07d | Helga Velroyen | Drawbacks of this design: |
238 | c2a3c07d | Helga Velroyen | |
239 | c2a3c07d | Helga Velroyen | - Complexity of node and certificate management will be increased (see |
240 | c2a3c07d | Helga Velroyen | following sections for details). |
241 | c2a3c07d | Helga Velroyen | - If the candidate map is not distributed fast enough to all nodes after |
242 | c2a3c07d | Helga Velroyen | an update of the configuration, it might be possible to issue RPC calls |
243 | c2a3c07d | Helga Velroyen | from a compromised master candidate node that has been removed |
244 | c2a3c07d | Helga Velroyen | from the Ganeti cluster already. However, this is still a better |
245 | c2a3c07d | Helga Velroyen | situation than before and an inherent problem when one wants to |
246 | c2a3c07d | Helga Velroyen | distinguish between master candidates and normal nodes. |
247 | c2a3c07d | Helga Velroyen | - A compromised master candidate would still be able to issue RPC calls, |
248 | c2a3c07d | Helga Velroyen | if it uses ssh to retrieve another master candidate's client |
249 | c2a3c07d | Helga Velroyen | certificate and the corresponding private SSL key. This is an issue |
250 | c2a3c07d | Helga Velroyen | even with the first part of the improved handling of ssh keys in this |
251 | c2a3c07d | Helga Velroyen | design (limiting ssh keys to master candidates), but it will be |
252 | c2a3c07d | Helga Velroyen | eliminated with the second part of the design (separate ssh keys for |
253 | c2a3c07d | Helga Velroyen | each master candidate). |
254 | 0565f862 | Helga Velroyen | - Even though this proposal is an improvement towards the previous |
255 | 0565f862 | Helga Velroyen | situation in Ganeti, it still does not use the full power of SSL. For |
256 | 0565f862 | Helga Velroyen | further improvements, see Section "Related and future work". |
257 | c2a3c07d | Helga Velroyen | |
258 | c2a3c07d | Helga Velroyen | Alternative proposals: |
259 | c2a3c07d | Helga Velroyen | |
260 | c2a3c07d | Helga Velroyen | - Instead of generating a client certificate per node, one could think |
261 | c2a3c07d | Helga Velroyen | of just generating two different client certificates, one for normal |
262 | c2a3c07d | Helga Velroyen | nodes and one for master candidates. Noded could then just check if |
263 | c2a3c07d | Helga Velroyen | the requesting node has the master candidate certificate. Drawback of |
264 | c2a3c07d | Helga Velroyen | this proposal is that once one master candidate gets compromised, all |
265 | c2a3c07d | Helga Velroyen | master candidates would need to get a new certificate even if the |
266 | c2a3c07d | Helga Velroyen | compromised master candidate had not yet fetched the certificates |
267 | c2a3c07d | Helga Velroyen | from the other master candidates via ssh. |
268 | c2a3c07d | Helga Velroyen | - In addition to our main proposal, one could think of including a |
269 | c2a3c07d | Helga Velroyen | piece of data (for example the node's host name or UUID) in the RPC |
270 | c2a3c07d | Helga Velroyen | call which is encrypted with the requesting node's private key. The |
271 | c2a3c07d | Helga Velroyen | node daemon could check if the datum can be decrypted using the node's |
272 | c2a3c07d | Helga Velroyen | certificate. However, this would ensure similar functionality as |
273 | c2a3c07d | Helga Velroyen | SSL's built-in client verification and add significant complexity |
274 | c2a3c07d | Helga Velroyen | to Ganeti's RPC protocol. |
275 | c2a3c07d | Helga Velroyen | |
276 | c2a3c07d | Helga Velroyen | In the following sections, we describe how our design affects various |
277 | c2a3c07d | Helga Velroyen | Ganeti operations. |
278 | c2a3c07d | Helga Velroyen | |
279 | c2a3c07d | Helga Velroyen | |
280 | c2a3c07d | Helga Velroyen | Cluster initialization |
281 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~ |
282 | c2a3c07d | Helga Velroyen | |
283 | c2a3c07d | Helga Velroyen | On cluster initialization, so far only the node daemon certificate was |
284 | c2a3c07d | Helga Velroyen | created. With our design, two certificates (and corresponding keys) |
285 | c2a3c07d | Helga Velroyen | need to be created, a server certificate to be distributed to all nodes |
286 | c2a3c07d | Helga Velroyen | and a client certificate only to be used by this particular node. In the |
287 | c2a3c07d | Helga Velroyen | following, we use the term node daemon certificate for the server |
288 | c2a3c07d | Helga Velroyen | certficate only. |
289 | c2a3c07d | Helga Velroyen | |
290 | c2a3c07d | Helga Velroyen | In the cluster configuration, the candidate map is created. It is |
291 | c2a3c07d | Helga Velroyen | populated with the respective entry for the master node. It is also |
292 | c2a3c07d | Helga Velroyen | written to ssconf. |
293 | c2a3c07d | Helga Velroyen | |
294 | c2a3c07d | Helga Velroyen | |
295 | c2a3c07d | Helga Velroyen | (Re-)Adding nodes |
296 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~ |
297 | c2a3c07d | Helga Velroyen | |
298 | c2a3c07d | Helga Velroyen | When a node is added, the server certificate is copied to the node (as |
299 | c2a3c07d | Helga Velroyen | before). Additionally, a new client certificate (and the corresponding |
300 | c2a3c07d | Helga Velroyen | private key) is created on the new node to be used only by the new node |
301 | c2a3c07d | Helga Velroyen | as client certifcate. |
302 | c2a3c07d | Helga Velroyen | |
303 | c2a3c07d | Helga Velroyen | If the new node is a master candidate, the candidate map is extended by |
304 | c2a3c07d | Helga Velroyen | the new node's data. As before, the updated configuration is distributed |
305 | c2a3c07d | Helga Velroyen | to all nodes (as complete configuration on the master candidates and |
306 | c2a3c07d | Helga Velroyen | ssconf on all nodes). Note that distribution of the configuration after |
307 | c2a3c07d | Helga Velroyen | adding a node is already implemented, since all nodes hold the list of |
308 | c2a3c07d | Helga Velroyen | nodes in the cluster in ssconf anyway. |
309 | c2a3c07d | Helga Velroyen | |
310 | c2a3c07d | Helga Velroyen | If the configuration for whatever reason already holds an entry for this |
311 | c2a3c07d | Helga Velroyen | node, it will be overriden. |
312 | c2a3c07d | Helga Velroyen | |
313 | c2a3c07d | Helga Velroyen | When readding a node, the procedure is the same as for adding a node. |
314 | c2a3c07d | Helga Velroyen | |
315 | c2a3c07d | Helga Velroyen | |
316 | c2a3c07d | Helga Velroyen | Promotion and demotion of master candidates |
317 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
318 | c2a3c07d | Helga Velroyen | |
319 | c2a3c07d | Helga Velroyen | When a normal node gets promoted to be master candidate, an entry to the |
320 | c2a3c07d | Helga Velroyen | candidate map has to be added and the updated configuration has to be |
321 | c2a3c07d | Helga Velroyen | distributed to all nodes. If there was already an entry for the node, |
322 | c2a3c07d | Helga Velroyen | we override it. |
323 | c2a3c07d | Helga Velroyen | |
324 | c2a3c07d | Helga Velroyen | On demotion of a master candidate, the node's entry in the candidate map |
325 | c2a3c07d | Helga Velroyen | gets removed and the updated configuration gets redistibuted. |
326 | c2a3c07d | Helga Velroyen | |
327 | c2a3c07d | Helga Velroyen | The same procedure applied to onlining and offlining master candidates. |
328 | c2a3c07d | Helga Velroyen | |
329 | c2a3c07d | Helga Velroyen | |
330 | c2a3c07d | Helga Velroyen | Cluster verify |
331 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~ |
332 | c2a3c07d | Helga Velroyen | |
333 | c2a3c07d | Helga Velroyen | Cluster verify will be extended by the following checks: |
334 | c2a3c07d | Helga Velroyen | |
335 | c2a3c07d | Helga Velroyen | - Whether each entry in the candidate map indeed corresponds to a master |
336 | c2a3c07d | Helga Velroyen | candidate. |
337 | c2a3c07d | Helga Velroyen | - Whether the master candidate's certificate digest match their entry |
338 | c2a3c07d | Helga Velroyen | in the candidate map. |
339 | 575b31bf | Helga Velroyen | - Whether no node tries to use the certificate of another node. In |
340 | 575b31bf | Helga Velroyen | particular, it is important to check that no normal node tries to |
341 | 575b31bf | Helga Velroyen | use the certificate of a master candidate. |
342 | c2a3c07d | Helga Velroyen | |
343 | c2a3c07d | Helga Velroyen | |
344 | c2a3c07d | Helga Velroyen | Crypto renewal |
345 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~ |
346 | c2a3c07d | Helga Velroyen | |
347 | c2a3c07d | Helga Velroyen | Currently, when the cluster's cryptographic tokens are renewed using the |
348 | c2a3c07d | Helga Velroyen | ``gnt-cluster renew-crypto`` command the node daemon certificate is |
349 | c2a3c07d | Helga Velroyen | renewed (among others). Option ``--new-cluster-certificate`` renews the |
350 | c2a3c07d | Helga Velroyen | node daemon certificate only. |
351 | c2a3c07d | Helga Velroyen | |
352 | c2a3c07d | Helga Velroyen | By adding an option ``--new-node-certificates`` we offer to renew the |
353 | 575b31bf | Helga Velroyen | client certificate. Whenever the client certificates are renewed, the |
354 | c2a3c07d | Helga Velroyen | candidate map has to be updated and redistributed. |
355 | c2a3c07d | Helga Velroyen | |
356 | 575b31bf | Helga Velroyen | If for whatever reason, the candidate map becomes inconsistent, for example |
357 | 575b31bf | Helga Velroyen | due inconsistent updating after a demotion or offlining), the user can use |
358 | 575b31bf | Helga Velroyen | this option to renew the client certificates and update the candidate |
359 | 575b31bf | Helga Velroyen | certificate map. |
360 | c2a3c07d | Helga Velroyen | |
361 | c2a3c07d | Helga Velroyen | |
362 | c2a3c07d | Helga Velroyen | Further considerations |
363 | c2a3c07d | Helga Velroyen | ---------------------- |
364 | c2a3c07d | Helga Velroyen | |
365 | c2a3c07d | Helga Velroyen | Watcher |
366 | c2a3c07d | Helga Velroyen | ~~~~~~~ |
367 | c2a3c07d | Helga Velroyen | |
368 | c2a3c07d | Helga Velroyen | The watcher is a script that is run on all nodes in regular intervals. The |
369 | c2a3c07d | Helga Velroyen | changes proposed in this design will not affect the watcher's implementation, |
370 | c2a3c07d | Helga Velroyen | because it behaves differently on the master than on non-master nodes. |
371 | c2a3c07d | Helga Velroyen | |
372 | c2a3c07d | Helga Velroyen | Only on the master, it issues query calls which would require a client |
373 | c2a3c07d | Helga Velroyen | certificate of a node in the candidate mapping. This is the case for the |
374 | c2a3c07d | Helga Velroyen | master node. On non-master node, it's only external communication is done via |
375 | c2a3c07d | Helga Velroyen | the ConfD protocol, which uses the hmac key, which is present on all nodes. |
376 | c2a3c07d | Helga Velroyen | Besides that, the watcher does not make any ssh connections, and thus is |
377 | c2a3c07d | Helga Velroyen | not affected by the changes in ssh key handling either. |
378 | c2a3c07d | Helga Velroyen | |
379 | c2a3c07d | Helga Velroyen | |
380 | c2a3c07d | Helga Velroyen | Other Keys and Daemons |
381 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~ |
382 | c2a3c07d | Helga Velroyen | |
383 | c2a3c07d | Helga Velroyen | Ganeti handles a couple of other keys/certificates that have not been mentioned |
384 | c2a3c07d | Helga Velroyen | in this design so far. Also, other daemons than the ones mentioned so far |
385 | c2a3c07d | Helga Velroyen | perform intra-cluster communication. Neither the keys nor the daemons will |
386 | c2a3c07d | Helga Velroyen | be affected by this design for several reasons: |
387 | c2a3c07d | Helga Velroyen | |
388 | 0565f862 | Helga Velroyen | - The hmac key used by ConfD (see :doc:`design-2.1`): the hmac key is still |
389 | c2a3c07d | Helga Velroyen | distributed to all nodes, because it was designed to be used for |
390 | c2a3c07d | Helga Velroyen | communicating with ConfD, which should be possible from all nodes. |
391 | c2a3c07d | Helga Velroyen | For example, the monitoring daemon which runs on all nodes uses it to |
392 | c2a3c07d | Helga Velroyen | retrieve information from ConfD. However, since communication with ConfD |
393 | c2a3c07d | Helga Velroyen | is read-only, a compromised node holding the hmac key does not enable an |
394 | c2a3c07d | Helga Velroyen | attacker to change the cluster's state. |
395 | c2a3c07d | Helga Velroyen | |
396 | c2a3c07d | Helga Velroyen | - The WConfD daemon writes the configuration to all master candidates |
397 | c2a3c07d | Helga Velroyen | via RPC. Since it only runs on the master node, it's ability to run |
398 | c2a3c07d | Helga Velroyen | RPC requests is maintained with this design. |
399 | c2a3c07d | Helga Velroyen | |
400 | c2a3c07d | Helga Velroyen | - The rapi SSL key certificate and rapi user/password file 'rapi_users' is |
401 | 0565f862 | Helga Velroyen | already only copied to the master candidates (see :doc:`design-2.1`, |
402 | c2a3c07d | Helga Velroyen | Section ``Redistribute Config``). |
403 | c2a3c07d | Helga Velroyen | |
404 | c2a3c07d | Helga Velroyen | - The spice certificates are still distributed to all nodes, since it should |
405 | c2a3c07d | Helga Velroyen | be possible to use spice to access VMs on any cluster node. |
406 | c2a3c07d | Helga Velroyen | |
407 | c2a3c07d | Helga Velroyen | - The cluster domain secret is used for inter-cluster instance moves. |
408 | c2a3c07d | Helga Velroyen | Since instances can be moved from any normal node of the source cluster to |
409 | c2a3c07d | Helga Velroyen | any normal node of the destination cluster, the presence of this |
410 | c2a3c07d | Helga Velroyen | secret on all nodes is necessary. |
411 | c2a3c07d | Helga Velroyen | |
412 | c2a3c07d | Helga Velroyen | |
413 | c2a3c07d | Helga Velroyen | Related and Future Work |
414 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~~ |
415 | c2a3c07d | Helga Velroyen | |
416 | 0565f862 | Helga Velroyen | There a couple of suggestions on how to improve the SSL setup even more. |
417 | 0565f862 | Helga Velroyen | As a trade-off wrt to complexity and implementation effort, we did not |
418 | 0565f862 | Helga Velroyen | implement them yet (as of version 2.11) but describe them here for |
419 | 0565f862 | Helga Velroyen | future reference. |
420 | 0565f862 | Helga Velroyen | |
421 | 0565f862 | Helga Velroyen | - All SSL certificates that Ganeti uses so far are self-signed. It would |
422 | 0565f862 | Helga Velroyen | increase the security if they were signed by a common CA. There is |
423 | 0565f862 | Helga Velroyen | already a design doc for a Ganeti CA which was suggested in a |
424 | 0565f862 | Helga Velroyen | different context (related to import/export). This would also be a |
425 | 0565f862 | Helga Velroyen | benefit for the RPC calls. See design doc :doc:`design-impexp2` for |
426 | 0565f862 | Helga Velroyen | more information. Implementing a CA is rather complex, because it |
427 | 0565f862 | Helga Velroyen | would mean also to support renewing the CA certificate and providing |
428 | 0565f862 | Helga Velroyen | and supporting infrastructure to revoke compromised certificates. |
429 | 0565f862 | Helga Velroyen | - An extension of the previous suggestion would be to even enable the |
430 | 0565f862 | Helga Velroyen | system administrator to use an external CA. Especially in bigger |
431 | 0565f862 | Helga Velroyen | setups, where already an SSL infrastructure exists, it would be useful |
432 | 0565f862 | Helga Velroyen | if Ganeti can simply be integrated with it, rather than forcing the |
433 | 0565f862 | Helga Velroyen | user to use the Ganeti CA. |
434 | 0565f862 | Helga Velroyen | - A lighter version of using a CA would be to use the server certificate |
435 | 0565f862 | Helga Velroyen | to sign the client certificate instead of using self-signed |
436 | 0565f862 | Helga Velroyen | certificates for both. The probleme here is that this would make |
437 | 0565f862 | Helga Velroyen | renewing the server certificate rather complicated, because all client |
438 | 0565f862 | Helga Velroyen | certificates would need to be resigned and redistributed as well, |
439 | 0565f862 | Helga Velroyen | which leads to interesting chicken-and-egg problems when this is done |
440 | 0565f862 | Helga Velroyen | via RPC calls. |
441 | 0565f862 | Helga Velroyen | - Ganeti RPC calls are currently done without checking if the hostname |
442 | 0565f862 | Helga Velroyen | of the node complies with the common name of the certificate. This |
443 | 0565f862 | Helga Velroyen | might be a desirable feature, but would increase the effort when a |
444 | 0565f862 | Helga Velroyen | node is renamed. |
445 | 0565f862 | Helga Velroyen | - The typical use case for SSL is to have one certificate per node |
446 | 0565f862 | Helga Velroyen | rather than one shared certificate (Ganeti's noded server certificate) |
447 | 0565f862 | Helga Velroyen | and a client certificate. One could change the design in a way that |
448 | 0565f862 | Helga Velroyen | only one certificate per node is used, but this would require a common |
449 | 0565f862 | Helga Velroyen | CA so that the validity of the certificate can be established by every |
450 | 0565f862 | Helga Velroyen | node in the cluster. |
451 | 0565f862 | Helga Velroyen | - With the proposed design, the serial numbers of the client |
452 | 0565f862 | Helga Velroyen | certificates are set to the node UUIDs. This is technically also not |
453 | 0565f862 | Helga Velroyen | complying to how SSL is supposed to be used, as the serial numbers |
454 | 0565f862 | Helga Velroyen | should reflect the enumeration of certificates created by the CA. Once |
455 | 0565f862 | Helga Velroyen | a CA is implemented, it might be reasonable to change this |
456 | 0565f862 | Helga Velroyen | accordingly. The implementation of the proposed design also has the |
457 | 0565f862 | Helga Velroyen | drawback of the serial number not changing even if the certificate is |
458 | 0565f862 | Helga Velroyen | replaced by a new one (for example when calling ``gnt-cluster renew- |
459 | 0565f862 | Helga Velroyen | crypt``), which also does not comply to way SSL was designed to be |
460 | 0565f862 | Helga Velroyen | used. |
461 | c2a3c07d | Helga Velroyen | |
462 | c2a3c07d | Helga Velroyen | .. vim: set textwidth=72 : |
463 | c2a3c07d | Helga Velroyen | .. Local Variables: |
464 | c2a3c07d | Helga Velroyen | .. mode: rst |
465 | c2a3c07d | Helga Velroyen | .. fill-column: 72 |
466 | c2a3c07d | Helga Velroyen | .. End: |