root / doc / design-node-security.rst @ 56c934da
History | View | Annotate | Download (17.6 kB)
1 | c2a3c07d | Helga Velroyen | ============================= |
---|---|---|---|
2 | c2a3c07d | Helga Velroyen | Improvements of Node Security |
3 | c2a3c07d | Helga Velroyen | ============================= |
4 | c2a3c07d | Helga Velroyen | |
5 | c2a3c07d | Helga Velroyen | This document describes an enhancement of Ganeti's security by restricting |
6 | c2a3c07d | Helga Velroyen | the distribution of security-sensitive data to the master and master |
7 | c2a3c07d | Helga Velroyen | candidates only. |
8 | c2a3c07d | Helga Velroyen | |
9 | c2a3c07d | Helga Velroyen | Note: In this document, we will use the term 'normal node' for a node that |
10 | c2a3c07d | Helga Velroyen | is neither master nor master-candidate. |
11 | c2a3c07d | Helga Velroyen | |
12 | c2a3c07d | Helga Velroyen | .. contents:: :depth: 4 |
13 | c2a3c07d | Helga Velroyen | |
14 | c2a3c07d | Helga Velroyen | Objective |
15 | c2a3c07d | Helga Velroyen | ========= |
16 | c2a3c07d | Helga Velroyen | |
17 | c2a3c07d | Helga Velroyen | Up till 2.10, Ganeti distributes security-relevant keys to all nodes, |
18 | c2a3c07d | Helga Velroyen | including nodes that are neither master nor master-candidates. Those |
19 | c2a3c07d | Helga Velroyen | keys are the private and public SSH keys for node communication and the |
20 | c2a3c07d | Helga Velroyen | SSL certficate and private key for RPC communication. Objective of this |
21 | c2a3c07d | Helga Velroyen | design is to limit the set of nodes that can establish ssh and RPC |
22 | c2a3c07d | Helga Velroyen | connections to the master and master candidates. |
23 | c2a3c07d | Helga Velroyen | |
24 | c2a3c07d | Helga Velroyen | As pointed out in |
25 | c2a3c07d | Helga Velroyen | `issue 377 <https://code.google.com/p/ganeti/issues/detail?id=377>`_, this |
26 | c2a3c07d | Helga Velroyen | is a security risk. Since all nodes have these keys, compromising |
27 | c2a3c07d | Helga Velroyen | any of those nodes would possibly give an attacker access to all other |
28 | c2a3c07d | Helga Velroyen | machines in the cluster. Reducing the set of nodes that are able to |
29 | c2a3c07d | Helga Velroyen | make ssh and RPC connections to the master and master candidates would |
30 | c2a3c07d | Helga Velroyen | significantly reduce the risk simply because fewer machines would be a |
31 | c2a3c07d | Helga Velroyen | valuable target for attackers. |
32 | c2a3c07d | Helga Velroyen | |
33 | c2a3c07d | Helga Velroyen | Note: For bigger installations of Ganeti, it is advisable to run master |
34 | c2a3c07d | Helga Velroyen | candidate nodes as non-vm-capable nodes. This would reduce the attack |
35 | c2a3c07d | Helga Velroyen | surface for the hypervisor exploitation. |
36 | c2a3c07d | Helga Velroyen | |
37 | c2a3c07d | Helga Velroyen | |
38 | c2a3c07d | Helga Velroyen | Detailed design |
39 | c2a3c07d | Helga Velroyen | =============== |
40 | c2a3c07d | Helga Velroyen | |
41 | c2a3c07d | Helga Velroyen | |
42 | c2a3c07d | Helga Velroyen | Current state and shortcomings |
43 | c2a3c07d | Helga Velroyen | ------------------------------ |
44 | c2a3c07d | Helga Velroyen | |
45 | c2a3c07d | Helga Velroyen | Currently (as of 2.10), all nodes hold the following information: |
46 | c2a3c07d | Helga Velroyen | |
47 | c2a3c07d | Helga Velroyen | - the ssh host keys (public and private) |
48 | c2a3c07d | Helga Velroyen | - the ssh root keys (public and private) |
49 | 575b31bf | Helga Velroyen | - node daemon certificate (the SSL client certificate and its |
50 | c2a3c07d | Helga Velroyen | corresponding private key) |
51 | c2a3c07d | Helga Velroyen | |
52 | c2a3c07d | Helga Velroyen | Concerning ssh, this setup contains the following security issue. Since |
53 | c2a3c07d | Helga Velroyen | all nodes of a cluster can ssh as root into any other cluster node, one |
54 | c2a3c07d | Helga Velroyen | compromised node can harm all other nodes of a cluster. |
55 | c2a3c07d | Helga Velroyen | |
56 | c2a3c07d | Helga Velroyen | Regarding the SSL encryption of the RPC communication with the node |
57 | c2a3c07d | Helga Velroyen | daemon, we currently have the following setup. There is only one |
58 | c2a3c07d | Helga Velroyen | certificate which is used as both, client and server certificate. Besides |
59 | c2a3c07d | Helga Velroyen | the SSL client verification, we check if the used client certificate is |
60 | c2a3c07d | Helga Velroyen | the same as the certificate stored on the server. |
61 | c2a3c07d | Helga Velroyen | |
62 | c2a3c07d | Helga Velroyen | This means that any node running a node daemon can also act as an RPC |
63 | c2a3c07d | Helga Velroyen | client and use it to issue RPC calls to other cluster nodes. This in |
64 | c2a3c07d | Helga Velroyen | turn means that any compromised node could be used to make RPC calls to |
65 | c2a3c07d | Helga Velroyen | any node (including itself) to gain full control over VMs. This could |
66 | c2a3c07d | Helga Velroyen | be used by an attacker to for example bring down the VMs or exploit bugs |
67 | c2a3c07d | Helga Velroyen | in the virtualization stacks to gain access to the host machines as well. |
68 | c2a3c07d | Helga Velroyen | |
69 | c2a3c07d | Helga Velroyen | |
70 | c2a3c07d | Helga Velroyen | Proposal concerning SSH key distribution |
71 | c2a3c07d | Helga Velroyen | ---------------------------------------- |
72 | c2a3c07d | Helga Velroyen | |
73 | c2a3c07d | Helga Velroyen | We propose two improvements regarding the ssh keys: |
74 | c2a3c07d | Helga Velroyen | |
75 | c2a3c07d | Helga Velroyen | #. Limit the distribution of the private ssh key to the master candidates. |
76 | c2a3c07d | Helga Velroyen | |
77 | c2a3c07d | Helga Velroyen | #. Use different ssh key pairs for each master candidate. |
78 | c2a3c07d | Helga Velroyen | |
79 | c2a3c07d | Helga Velroyen | We propose to limit the set of nodes holding the private root user SSH key |
80 | c2a3c07d | Helga Velroyen | to the master and the master candidates. This way, the security risk would |
81 | c2a3c07d | Helga Velroyen | be limited to a rather small set of nodes even though the cluster could |
82 | c2a3c07d | Helga Velroyen | consists of a lot more nodes. The set of master candidates could be protected |
83 | c2a3c07d | Helga Velroyen | better than the normal nodes (for example residing in a DMZ) to enhance |
84 | c2a3c07d | Helga Velroyen | security even more if the administrator wishes so. The following |
85 | c2a3c07d | Helga Velroyen | sections describe in detail which Ganeti commands are affected by this |
86 | c2a3c07d | Helga Velroyen | change and in what way. |
87 | c2a3c07d | Helga Velroyen | |
88 | c2a3c07d | Helga Velroyen | Security will be even more increased if each master candidate gets |
89 | c2a3c07d | Helga Velroyen | its own ssh private/public key pair. This way, one can remove a |
90 | c2a3c07d | Helga Velroyen | compromised master candidate from a cluster (including removing it's |
91 | c2a3c07d | Helga Velroyen | public key from all nodes' ``authorized_keys`` file) without having to |
92 | c2a3c07d | Helga Velroyen | regenerate and distribute new ssh keys for all master candidates. (Even |
93 | c2a3c07d | Helga Velroyen | though it is be good practice to do that anyway, since the compromising |
94 | c2a3c07d | Helga Velroyen | of the other master candidates might have taken place already.) However, |
95 | c2a3c07d | Helga Velroyen | this improvement was not part of the original feature request and |
96 | c2a3c07d | Helga Velroyen | increases the complexity of node management even more. We therefore |
97 | c2a3c07d | Helga Velroyen | consider it as second step in this design and will address |
98 | c2a3c07d | Helga Velroyen | this after the other parts of this design are implemented. |
99 | c2a3c07d | Helga Velroyen | |
100 | c2a3c07d | Helga Velroyen | The following sections describe in detail which Ganeti commands are affected |
101 | c2a3c07d | Helga Velroyen | by the first part of ssh-related improvements, limiting the key |
102 | c2a3c07d | Helga Velroyen | distribution to master candidates only. |
103 | c2a3c07d | Helga Velroyen | |
104 | c2a3c07d | Helga Velroyen | |
105 | c2a3c07d | Helga Velroyen | (Re-)Adding nodes to a cluster |
106 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
107 | c2a3c07d | Helga Velroyen | |
108 | c2a3c07d | Helga Velroyen | According to ``design-node-add.rst``, Ganeti transfers the ssh keys to every |
109 | c2a3c07d | Helga Velroyen | node that gets added to the cluster. |
110 | c2a3c07d | Helga Velroyen | |
111 | c2a3c07d | Helga Velroyen | We propose to change this procedure to treat master candidates and normal |
112 | c2a3c07d | Helga Velroyen | nodes differently. For master candidates, the procedure would stay as is. |
113 | c2a3c07d | Helga Velroyen | For normal nodes, Ganeti would transfer the public and private ssh host |
114 | c2a3c07d | Helga Velroyen | keys (as before) and only the public root key. |
115 | c2a3c07d | Helga Velroyen | |
116 | c2a3c07d | Helga Velroyen | A normal node would not be able to connect via ssh to other nodes, but |
117 | c2a3c07d | Helga Velroyen | the master (and potentially master candidates) can connect to this node. |
118 | c2a3c07d | Helga Velroyen | |
119 | c2a3c07d | Helga Velroyen | In case of readding a node that used to be in the cluster before, |
120 | c2a3c07d | Helga Velroyen | handling of the ssh keys would basically be the same with the following |
121 | c2a3c07d | Helga Velroyen | additional modifications: if the node used to be a master or |
122 | c2a3c07d | Helga Velroyen | master-candidate node, but will be a normal node after readding, Ganeti |
123 | c2a3c07d | Helga Velroyen | should make sure that the private root key is deleted if it is still |
124 | c2a3c07d | Helga Velroyen | present on the node. |
125 | c2a3c07d | Helga Velroyen | |
126 | c2a3c07d | Helga Velroyen | |
127 | c2a3c07d | Helga Velroyen | Pro- and demoting a node to/from master candidate |
128 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
129 | c2a3c07d | Helga Velroyen | |
130 | c2a3c07d | Helga Velroyen | If the role of a node is changed from 'normal' to 'master_candidate', the |
131 | c2a3c07d | Helga Velroyen | master node should at that point copy the private root ssh key. When demoting |
132 | c2a3c07d | Helga Velroyen | a node from master candidate to a normal node, the key that have been copied |
133 | c2a3c07d | Helga Velroyen | there on promotion or addition should be removed again. |
134 | c2a3c07d | Helga Velroyen | |
135 | c2a3c07d | Helga Velroyen | This affected the behavior of the following commands: |
136 | c2a3c07d | Helga Velroyen | |
137 | c2a3c07d | Helga Velroyen | :: |
138 | c2a3c07d | Helga Velroyen | gnt-node modify --master-candidate=yes |
139 | c2a3c07d | Helga Velroyen | gnt-node modify --master-candidate=no [--auto-promote] |
140 | c2a3c07d | Helga Velroyen | |
141 | c2a3c07d | Helga Velroyen | If the node has been master candidate already before the command to promote |
142 | c2a3c07d | Helga Velroyen | it was issued, Ganeti does not do anything. |
143 | c2a3c07d | Helga Velroyen | |
144 | c2a3c07d | Helga Velroyen | Note that when you demote a node from master candidate to normal node, another |
145 | c2a3c07d | Helga Velroyen | master-capable and normal node will be promoted to master candidate. For this |
146 | c2a3c07d | Helga Velroyen | newly promoted node, the same changes apply as if it was explicitely promoted. |
147 | c2a3c07d | Helga Velroyen | |
148 | c2a3c07d | Helga Velroyen | The same behavior should be ensured for the corresponding rapi command. |
149 | c2a3c07d | Helga Velroyen | |
150 | c2a3c07d | Helga Velroyen | |
151 | c2a3c07d | Helga Velroyen | Offlining and onlining a node |
152 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
153 | c2a3c07d | Helga Velroyen | |
154 | c2a3c07d | Helga Velroyen | When offlining a node, it immediately loses its role as master or master |
155 | c2a3c07d | Helga Velroyen | candidate as well. When it is onlined again, it will become master |
156 | c2a3c07d | Helga Velroyen | candidate again if it was so before. The handling of the keys should be done |
157 | c2a3c07d | Helga Velroyen | in the same way as when the node is explicitely promoted or demoted to or from |
158 | c2a3c07d | Helga Velroyen | master candidate. See the previous section for details. |
159 | c2a3c07d | Helga Velroyen | |
160 | c2a3c07d | Helga Velroyen | This affects the command: |
161 | c2a3c07d | Helga Velroyen | |
162 | c2a3c07d | Helga Velroyen | :: |
163 | c2a3c07d | Helga Velroyen | gnt-node modify --offline=yes |
164 | c2a3c07d | Helga Velroyen | gnt-node modify --offline=no [--auto-promote] |
165 | c2a3c07d | Helga Velroyen | |
166 | c2a3c07d | Helga Velroyen | For offlining, the removal of the keys is particularly important, as the |
167 | c2a3c07d | Helga Velroyen | detection of a compromised node might be the very reason for the offlining. |
168 | c2a3c07d | Helga Velroyen | Of course we cannot guarantee that removal of the key is always successful, |
169 | c2a3c07d | Helga Velroyen | because the node might not be reachable anymore. Even though it is a |
170 | c2a3c07d | Helga Velroyen | best-effort operation, it is still an improvement over the status quo, |
171 | c2a3c07d | Helga Velroyen | because currently Ganeti does not even try to remove any keys. |
172 | c2a3c07d | Helga Velroyen | |
173 | c2a3c07d | Helga Velroyen | The same behavior should be ensured for the corresponding rapi command. |
174 | c2a3c07d | Helga Velroyen | |
175 | c2a3c07d | Helga Velroyen | |
176 | c2a3c07d | Helga Velroyen | Cluster verify |
177 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~ |
178 | c2a3c07d | Helga Velroyen | |
179 | c2a3c07d | Helga Velroyen | To make sure the private root ssh key was not distributed to a normal |
180 | c2a3c07d | Helga Velroyen | node, 'gnt-cluster verify' will be extended by a check for the key |
181 | c2a3c07d | Helga Velroyen | on normal nodes. Additionally, it will check if the private key is |
182 | c2a3c07d | Helga Velroyen | indeed present on master candidates. |
183 | c2a3c07d | Helga Velroyen | |
184 | c2a3c07d | Helga Velroyen | |
185 | c2a3c07d | Helga Velroyen | |
186 | c2a3c07d | Helga Velroyen | Proposal regarding node daemon certificates |
187 | c2a3c07d | Helga Velroyen | ------------------------------------------- |
188 | c2a3c07d | Helga Velroyen | |
189 | c2a3c07d | Helga Velroyen | Regarding the node daemon certificates, we propose the following changes |
190 | c2a3c07d | Helga Velroyen | in the design. |
191 | c2a3c07d | Helga Velroyen | |
192 | c2a3c07d | Helga Velroyen | - Instead of using the same certificate for all nodes as both, server |
193 | c2a3c07d | Helga Velroyen | and client certificate, we generate a common server certificate (and |
194 | c2a3c07d | Helga Velroyen | the corresponding private key) for all nodes and a different client |
195 | c2a3c07d | Helga Velroyen | certificate (and the corresponding private key) for each node. |
196 | c2a3c07d | Helga Velroyen | - In addition, we store a mapping of |
197 | c2a3c07d | Helga Velroyen | (node UUID, client certificate digest) in the cluster's configuration |
198 | c2a3c07d | Helga Velroyen | and ssconf for hosts that are master or master candidate. |
199 | c2a3c07d | Helga Velroyen | The client certificate digest is a hash of the client certificate. |
200 | c2a3c07d | Helga Velroyen | We suggest a 'sha1' hash here. We will call this mapping 'candidate map' |
201 | c2a3c07d | Helga Velroyen | from here on. |
202 | c2a3c07d | Helga Velroyen | - The node daemon will be modified in a way that on an incoming RPC |
203 | c2a3c07d | Helga Velroyen | request, it first performs a client verification (same as before) to |
204 | c2a3c07d | Helga Velroyen | ensure that the requesting host is indeed the holder of the |
205 | c2a3c07d | Helga Velroyen | corresponding private key. Additionally, it compares the digest of |
206 | c2a3c07d | Helga Velroyen | the certificate of the incoming request to the respective entry of |
207 | c2a3c07d | Helga Velroyen | the candidate map. If the digest does not match the entry of the host |
208 | c2a3c07d | Helga Velroyen | in the mapping or is not included in the mapping at all, the SSL |
209 | c2a3c07d | Helga Velroyen | connection is refused. |
210 | c2a3c07d | Helga Velroyen | |
211 | c2a3c07d | Helga Velroyen | This design has the following advantages: |
212 | c2a3c07d | Helga Velroyen | |
213 | c2a3c07d | Helga Velroyen | - A compromised normal node cannot issue RPC calls, because it will |
214 | c2a3c07d | Helga Velroyen | not be in the candidate map. (See the ``Drawbacks`` section regarding |
215 | c2a3c07d | Helga Velroyen | an indirect way of achieving this though.) |
216 | c2a3c07d | Helga Velroyen | - A compromised master candidate would be able to issue RPC requests, |
217 | c2a3c07d | Helga Velroyen | but on detection of its compromised state, it can be removed from the |
218 | c2a3c07d | Helga Velroyen | cluster (and thus from the candidate map) without the need for |
219 | c2a3c07d | Helga Velroyen | redistribution of any certificates, because the other master candidates |
220 | c2a3c07d | Helga Velroyen | can continue using their own certificates. However, it is best |
221 | c2a3c07d | Helga Velroyen | practice to issue a complete key renewal even in this case, unless one |
222 | c2a3c07d | Helga Velroyen | can ensure no actions compromising other nodes have not already been |
223 | c2a3c07d | Helga Velroyen | carried out. |
224 | c2a3c07d | Helga Velroyen | - A compromised node would not be able to use the other (possibly master |
225 | c2a3c07d | Helga Velroyen | candidate) nodes' information from the candidate map to issue RPCs, |
226 | c2a3c07d | Helga Velroyen | because the config just stores the digests and not the certificate |
227 | c2a3c07d | Helga Velroyen | itself. |
228 | c2a3c07d | Helga Velroyen | - A compromised node would be able to obtain another node's certificate |
229 | c2a3c07d | Helga Velroyen | by waiting for incoming RPCs from this other node. However, the node |
230 | c2a3c07d | Helga Velroyen | cannot use the certificate to issue RPC calls, because the SSL client |
231 | c2a3c07d | Helga Velroyen | verification would require the node to hold the corresponding private |
232 | c2a3c07d | Helga Velroyen | key as well. |
233 | c2a3c07d | Helga Velroyen | |
234 | c2a3c07d | Helga Velroyen | Drawbacks of this design: |
235 | c2a3c07d | Helga Velroyen | |
236 | c2a3c07d | Helga Velroyen | - Complexity of node and certificate management will be increased (see |
237 | c2a3c07d | Helga Velroyen | following sections for details). |
238 | c2a3c07d | Helga Velroyen | - If the candidate map is not distributed fast enough to all nodes after |
239 | c2a3c07d | Helga Velroyen | an update of the configuration, it might be possible to issue RPC calls |
240 | c2a3c07d | Helga Velroyen | from a compromised master candidate node that has been removed |
241 | c2a3c07d | Helga Velroyen | from the Ganeti cluster already. However, this is still a better |
242 | c2a3c07d | Helga Velroyen | situation than before and an inherent problem when one wants to |
243 | c2a3c07d | Helga Velroyen | distinguish between master candidates and normal nodes. |
244 | c2a3c07d | Helga Velroyen | - A compromised master candidate would still be able to issue RPC calls, |
245 | c2a3c07d | Helga Velroyen | if it uses ssh to retrieve another master candidate's client |
246 | c2a3c07d | Helga Velroyen | certificate and the corresponding private SSL key. This is an issue |
247 | c2a3c07d | Helga Velroyen | even with the first part of the improved handling of ssh keys in this |
248 | c2a3c07d | Helga Velroyen | design (limiting ssh keys to master candidates), but it will be |
249 | c2a3c07d | Helga Velroyen | eliminated with the second part of the design (separate ssh keys for |
250 | c2a3c07d | Helga Velroyen | each master candidate). |
251 | c2a3c07d | Helga Velroyen | |
252 | c2a3c07d | Helga Velroyen | Alternative proposals: |
253 | c2a3c07d | Helga Velroyen | |
254 | c2a3c07d | Helga Velroyen | - Instead of generating a client certificate per node, one could think |
255 | c2a3c07d | Helga Velroyen | of just generating two different client certificates, one for normal |
256 | c2a3c07d | Helga Velroyen | nodes and one for master candidates. Noded could then just check if |
257 | c2a3c07d | Helga Velroyen | the requesting node has the master candidate certificate. Drawback of |
258 | c2a3c07d | Helga Velroyen | this proposal is that once one master candidate gets compromised, all |
259 | c2a3c07d | Helga Velroyen | master candidates would need to get a new certificate even if the |
260 | c2a3c07d | Helga Velroyen | compromised master candidate had not yet fetched the certificates |
261 | c2a3c07d | Helga Velroyen | from the other master candidates via ssh. |
262 | c2a3c07d | Helga Velroyen | - In addition to our main proposal, one could think of including a |
263 | c2a3c07d | Helga Velroyen | piece of data (for example the node's host name or UUID) in the RPC |
264 | c2a3c07d | Helga Velroyen | call which is encrypted with the requesting node's private key. The |
265 | c2a3c07d | Helga Velroyen | node daemon could check if the datum can be decrypted using the node's |
266 | c2a3c07d | Helga Velroyen | certificate. However, this would ensure similar functionality as |
267 | c2a3c07d | Helga Velroyen | SSL's built-in client verification and add significant complexity |
268 | c2a3c07d | Helga Velroyen | to Ganeti's RPC protocol. |
269 | c2a3c07d | Helga Velroyen | |
270 | c2a3c07d | Helga Velroyen | In the following sections, we describe how our design affects various |
271 | c2a3c07d | Helga Velroyen | Ganeti operations. |
272 | c2a3c07d | Helga Velroyen | |
273 | c2a3c07d | Helga Velroyen | |
274 | c2a3c07d | Helga Velroyen | Cluster initialization |
275 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~ |
276 | c2a3c07d | Helga Velroyen | |
277 | c2a3c07d | Helga Velroyen | On cluster initialization, so far only the node daemon certificate was |
278 | c2a3c07d | Helga Velroyen | created. With our design, two certificates (and corresponding keys) |
279 | c2a3c07d | Helga Velroyen | need to be created, a server certificate to be distributed to all nodes |
280 | c2a3c07d | Helga Velroyen | and a client certificate only to be used by this particular node. In the |
281 | c2a3c07d | Helga Velroyen | following, we use the term node daemon certificate for the server |
282 | c2a3c07d | Helga Velroyen | certficate only. |
283 | c2a3c07d | Helga Velroyen | |
284 | c2a3c07d | Helga Velroyen | In the cluster configuration, the candidate map is created. It is |
285 | c2a3c07d | Helga Velroyen | populated with the respective entry for the master node. It is also |
286 | c2a3c07d | Helga Velroyen | written to ssconf. |
287 | c2a3c07d | Helga Velroyen | |
288 | c2a3c07d | Helga Velroyen | |
289 | c2a3c07d | Helga Velroyen | (Re-)Adding nodes |
290 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~ |
291 | c2a3c07d | Helga Velroyen | |
292 | c2a3c07d | Helga Velroyen | When a node is added, the server certificate is copied to the node (as |
293 | c2a3c07d | Helga Velroyen | before). Additionally, a new client certificate (and the corresponding |
294 | c2a3c07d | Helga Velroyen | private key) is created on the new node to be used only by the new node |
295 | c2a3c07d | Helga Velroyen | as client certifcate. |
296 | c2a3c07d | Helga Velroyen | |
297 | c2a3c07d | Helga Velroyen | If the new node is a master candidate, the candidate map is extended by |
298 | c2a3c07d | Helga Velroyen | the new node's data. As before, the updated configuration is distributed |
299 | c2a3c07d | Helga Velroyen | to all nodes (as complete configuration on the master candidates and |
300 | c2a3c07d | Helga Velroyen | ssconf on all nodes). Note that distribution of the configuration after |
301 | c2a3c07d | Helga Velroyen | adding a node is already implemented, since all nodes hold the list of |
302 | c2a3c07d | Helga Velroyen | nodes in the cluster in ssconf anyway. |
303 | c2a3c07d | Helga Velroyen | |
304 | c2a3c07d | Helga Velroyen | If the configuration for whatever reason already holds an entry for this |
305 | c2a3c07d | Helga Velroyen | node, it will be overriden. |
306 | c2a3c07d | Helga Velroyen | |
307 | c2a3c07d | Helga Velroyen | When readding a node, the procedure is the same as for adding a node. |
308 | c2a3c07d | Helga Velroyen | |
309 | c2a3c07d | Helga Velroyen | |
310 | c2a3c07d | Helga Velroyen | Promotion and demotion of master candidates |
311 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
312 | c2a3c07d | Helga Velroyen | |
313 | c2a3c07d | Helga Velroyen | When a normal node gets promoted to be master candidate, an entry to the |
314 | c2a3c07d | Helga Velroyen | candidate map has to be added and the updated configuration has to be |
315 | c2a3c07d | Helga Velroyen | distributed to all nodes. If there was already an entry for the node, |
316 | c2a3c07d | Helga Velroyen | we override it. |
317 | c2a3c07d | Helga Velroyen | |
318 | c2a3c07d | Helga Velroyen | On demotion of a master candidate, the node's entry in the candidate map |
319 | c2a3c07d | Helga Velroyen | gets removed and the updated configuration gets redistibuted. |
320 | c2a3c07d | Helga Velroyen | |
321 | c2a3c07d | Helga Velroyen | The same procedure applied to onlining and offlining master candidates. |
322 | c2a3c07d | Helga Velroyen | |
323 | c2a3c07d | Helga Velroyen | |
324 | c2a3c07d | Helga Velroyen | Cluster verify |
325 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~ |
326 | c2a3c07d | Helga Velroyen | |
327 | c2a3c07d | Helga Velroyen | Cluster verify will be extended by the following checks: |
328 | c2a3c07d | Helga Velroyen | |
329 | c2a3c07d | Helga Velroyen | - Whether each entry in the candidate map indeed corresponds to a master |
330 | c2a3c07d | Helga Velroyen | candidate. |
331 | c2a3c07d | Helga Velroyen | - Whether the master candidate's certificate digest match their entry |
332 | c2a3c07d | Helga Velroyen | in the candidate map. |
333 | 575b31bf | Helga Velroyen | - Whether no node tries to use the certificate of another node. In |
334 | 575b31bf | Helga Velroyen | particular, it is important to check that no normal node tries to |
335 | 575b31bf | Helga Velroyen | use the certificate of a master candidate. |
336 | c2a3c07d | Helga Velroyen | |
337 | c2a3c07d | Helga Velroyen | |
338 | c2a3c07d | Helga Velroyen | Crypto renewal |
339 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~ |
340 | c2a3c07d | Helga Velroyen | |
341 | c2a3c07d | Helga Velroyen | Currently, when the cluster's cryptographic tokens are renewed using the |
342 | c2a3c07d | Helga Velroyen | ``gnt-cluster renew-crypto`` command the node daemon certificate is |
343 | c2a3c07d | Helga Velroyen | renewed (among others). Option ``--new-cluster-certificate`` renews the |
344 | c2a3c07d | Helga Velroyen | node daemon certificate only. |
345 | c2a3c07d | Helga Velroyen | |
346 | c2a3c07d | Helga Velroyen | By adding an option ``--new-node-certificates`` we offer to renew the |
347 | 575b31bf | Helga Velroyen | client certificate. Whenever the client certificates are renewed, the |
348 | c2a3c07d | Helga Velroyen | candidate map has to be updated and redistributed. |
349 | c2a3c07d | Helga Velroyen | |
350 | 575b31bf | Helga Velroyen | If for whatever reason, the candidate map becomes inconsistent, for example |
351 | 575b31bf | Helga Velroyen | due inconsistent updating after a demotion or offlining), the user can use |
352 | 575b31bf | Helga Velroyen | this option to renew the client certificates and update the candidate |
353 | 575b31bf | Helga Velroyen | certificate map. |
354 | c2a3c07d | Helga Velroyen | |
355 | c2a3c07d | Helga Velroyen | |
356 | c2a3c07d | Helga Velroyen | Further considerations |
357 | c2a3c07d | Helga Velroyen | ---------------------- |
358 | c2a3c07d | Helga Velroyen | |
359 | c2a3c07d | Helga Velroyen | Watcher |
360 | c2a3c07d | Helga Velroyen | ~~~~~~~ |
361 | c2a3c07d | Helga Velroyen | |
362 | c2a3c07d | Helga Velroyen | The watcher is a script that is run on all nodes in regular intervals. The |
363 | c2a3c07d | Helga Velroyen | changes proposed in this design will not affect the watcher's implementation, |
364 | c2a3c07d | Helga Velroyen | because it behaves differently on the master than on non-master nodes. |
365 | c2a3c07d | Helga Velroyen | |
366 | c2a3c07d | Helga Velroyen | Only on the master, it issues query calls which would require a client |
367 | c2a3c07d | Helga Velroyen | certificate of a node in the candidate mapping. This is the case for the |
368 | c2a3c07d | Helga Velroyen | master node. On non-master node, it's only external communication is done via |
369 | c2a3c07d | Helga Velroyen | the ConfD protocol, which uses the hmac key, which is present on all nodes. |
370 | c2a3c07d | Helga Velroyen | Besides that, the watcher does not make any ssh connections, and thus is |
371 | c2a3c07d | Helga Velroyen | not affected by the changes in ssh key handling either. |
372 | c2a3c07d | Helga Velroyen | |
373 | c2a3c07d | Helga Velroyen | |
374 | c2a3c07d | Helga Velroyen | Other Keys and Daemons |
375 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~ |
376 | c2a3c07d | Helga Velroyen | |
377 | c2a3c07d | Helga Velroyen | Ganeti handles a couple of other keys/certificates that have not been mentioned |
378 | c2a3c07d | Helga Velroyen | in this design so far. Also, other daemons than the ones mentioned so far |
379 | c2a3c07d | Helga Velroyen | perform intra-cluster communication. Neither the keys nor the daemons will |
380 | c2a3c07d | Helga Velroyen | be affected by this design for several reasons: |
381 | c2a3c07d | Helga Velroyen | |
382 | c2a3c07d | Helga Velroyen | - The hmac key used by ConfD (see ``design-2.1.rst``): the hmac key is still |
383 | c2a3c07d | Helga Velroyen | distributed to all nodes, because it was designed to be used for |
384 | c2a3c07d | Helga Velroyen | communicating with ConfD, which should be possible from all nodes. |
385 | c2a3c07d | Helga Velroyen | For example, the monitoring daemon which runs on all nodes uses it to |
386 | c2a3c07d | Helga Velroyen | retrieve information from ConfD. However, since communication with ConfD |
387 | c2a3c07d | Helga Velroyen | is read-only, a compromised node holding the hmac key does not enable an |
388 | c2a3c07d | Helga Velroyen | attacker to change the cluster's state. |
389 | c2a3c07d | Helga Velroyen | |
390 | c2a3c07d | Helga Velroyen | - The WConfD daemon writes the configuration to all master candidates |
391 | c2a3c07d | Helga Velroyen | via RPC. Since it only runs on the master node, it's ability to run |
392 | c2a3c07d | Helga Velroyen | RPC requests is maintained with this design. |
393 | c2a3c07d | Helga Velroyen | |
394 | c2a3c07d | Helga Velroyen | - The rapi SSL key certificate and rapi user/password file 'rapi_users' is |
395 | c2a3c07d | Helga Velroyen | already only copied to the master candidates (see ``design-2.1.rst``, |
396 | c2a3c07d | Helga Velroyen | Section ``Redistribute Config``). |
397 | c2a3c07d | Helga Velroyen | |
398 | c2a3c07d | Helga Velroyen | - The spice certificates are still distributed to all nodes, since it should |
399 | c2a3c07d | Helga Velroyen | be possible to use spice to access VMs on any cluster node. |
400 | c2a3c07d | Helga Velroyen | |
401 | c2a3c07d | Helga Velroyen | - The cluster domain secret is used for inter-cluster instance moves. |
402 | c2a3c07d | Helga Velroyen | Since instances can be moved from any normal node of the source cluster to |
403 | c2a3c07d | Helga Velroyen | any normal node of the destination cluster, the presence of this |
404 | c2a3c07d | Helga Velroyen | secret on all nodes is necessary. |
405 | c2a3c07d | Helga Velroyen | |
406 | c2a3c07d | Helga Velroyen | |
407 | c2a3c07d | Helga Velroyen | Related and Future Work |
408 | c2a3c07d | Helga Velroyen | ~~~~~~~~~~~~~~~~~~~~~~~ |
409 | c2a3c07d | Helga Velroyen | |
410 | c2a3c07d | Helga Velroyen | Ganeti RPC calls are currently done without server verification. |
411 | c2a3c07d | Helga Velroyen | Establishing server verification might be a desirable feature, but is |
412 | c2a3c07d | Helga Velroyen | not part of this design. |
413 | c2a3c07d | Helga Velroyen | |
414 | c2a3c07d | Helga Velroyen | .. vim: set textwidth=72 : |
415 | c2a3c07d | Helga Velroyen | .. Local Variables: |
416 | c2a3c07d | Helga Velroyen | .. mode: rst |
417 | c2a3c07d | Helga Velroyen | .. fill-column: 72 |
418 | c2a3c07d | Helga Velroyen | .. End: |