root / doc / design-network.rst @ a17deeab
History | View | Annotate | Download (10.1 kB)
1 |
================== |
---|---|
2 |
Network management |
3 |
================== |
4 |
|
5 |
.. contents:: :depth: 4 |
6 |
|
7 |
This is a design document detailing the implementation of network resource |
8 |
management in Ganeti. |
9 |
|
10 |
Current state and shortcomings |
11 |
============================== |
12 |
|
13 |
Currently Ganeti supports two configuration modes for instance NICs: |
14 |
routed and bridged mode. The ``ip`` NIC parameter, which is mandatory |
15 |
for routed NICs and optional for bridged ones, holds the given NIC's IP |
16 |
address and may be filled either manually, or via a DNS lookup for the |
17 |
instance's hostname. |
18 |
|
19 |
This approach presents some shortcomings: |
20 |
|
21 |
a) It relies on external systems to perform network resource |
22 |
management. Although large organizations may already have IP pool |
23 |
management software in place, this is not usually the case with |
24 |
stand-alone deployments. For smaller installations it makes sense to |
25 |
allocate a pool of IP addresses to Ganeti and let it transparently |
26 |
assign these IPs to instances as appropriate. |
27 |
|
28 |
b) The NIC network information is incomplete, lacking netmask and |
29 |
gateway. Operating system providers could for example use the |
30 |
complete network information to fully configure an instance's |
31 |
network parameters upon its creation. |
32 |
|
33 |
Furthermore, having full network configuration information would |
34 |
enable Ganeti nodes to become more self-contained and be able to |
35 |
infer system configuration (e.g. /etc/network/interfaces content) |
36 |
from Ganeti configuration. This should make configuration of |
37 |
newly-added nodes a lot easier and less dependant on external |
38 |
tools/procedures. |
39 |
|
40 |
c) Instance placement must explicitly take network availability in |
41 |
different node groups into account; the same ``link`` is implicitly |
42 |
expected to connect to the same network across the whole cluster, |
43 |
which may not always be the case with large clusters with multiple |
44 |
node groups. |
45 |
|
46 |
|
47 |
Proposed changes |
48 |
---------------- |
49 |
|
50 |
In order to deal with the above shortcomings, we propose to extend |
51 |
Ganeti with high-level network management logic, which consists of a new |
52 |
NIC mode called ``managed``, a new "Network" configuration object and |
53 |
logic to perform IP address pool management, i.e. maintain a set of |
54 |
available and occupied IP addresses. |
55 |
|
56 |
Configuration changes |
57 |
+++++++++++++++++++++ |
58 |
|
59 |
We propose the introduction of a new high-level Network object, |
60 |
containing (at least) the following data: |
61 |
|
62 |
- Symbolic name |
63 |
- UUID |
64 |
- Network in CIDR notation (IPv4 + IPv6) |
65 |
- Default gateway, if one exists (IPv4 + IPv6) |
66 |
- IP pool management data (reservations) |
67 |
- Default NIC connectivity mode (bridged, routed). This is the |
68 |
functional equivalent of the current NIC ``mode``. |
69 |
- Default host interface (e.g. br0). This is the functional equivalent |
70 |
of the current NIC ``link``. |
71 |
- Tags |
72 |
|
73 |
Each network will be connected to any number of node groups, possibly |
74 |
overriding connectivity mode and host interface for each node group. |
75 |
This is achieved by adding a ``networks`` slot to the NodeGroup object |
76 |
and using the networks' UUIDs as keys. |
77 |
|
78 |
IP pool management |
79 |
++++++++++++++++++ |
80 |
|
81 |
A new helper library is introduced, wrapping around Network objects to |
82 |
give IP pool management capabilities. A network's pool is defined by two |
83 |
bitfields, the length of the network size each: |
84 |
|
85 |
``reservations`` |
86 |
This field holds all IP addresses reserved by Ganeti instances, as |
87 |
well as cluster IP addresses (node addresses + cluster master) |
88 |
|
89 |
``external reservations`` |
90 |
This field holds all IP addresses that are manually reserved by the |
91 |
administrator, because some other equipment is using them outside the |
92 |
scope of Ganeti. |
93 |
|
94 |
The bitfields are implemented using the python-bitarray package for |
95 |
space efficiency and their binary value stored base64-encoded for JSON |
96 |
compatibility. This approach gives relatively compact representations |
97 |
even for large IPv4 networks (e.g. /20). |
98 |
|
99 |
Ganeti-owned IP addresses (node + master IPs) are reserved automatically |
100 |
if the cluster's data network itself is placed under pool management. |
101 |
|
102 |
Helper ConfigWriter methods provide free IP address generation and |
103 |
reservation, using a TemporaryReservationManager. |
104 |
|
105 |
It should be noted that IP pool management is performed only for IPv4 |
106 |
networks, as they are expected to be densely populated. IPv6 networks |
107 |
can use different approaches, e.g. sequential address asignment or |
108 |
EUI-64 addresses. |
109 |
|
110 |
Managed NIC mode |
111 |
++++++++++++++++ |
112 |
|
113 |
In order to be able to use the new network facility while maintaining |
114 |
compatibility with the current networking model, a new network mode is |
115 |
introduced, called ``managed`` to reflect the fact that the given NICs |
116 |
network configuration is managed by Ganeti itself. A managed mode NIC |
117 |
accepts the network it is connected to in its ``link`` argument. |
118 |
Userspace tools can refer to networks using their symbolic names, |
119 |
however internally, the link argument stores the network's UUID. |
120 |
|
121 |
We also introduce a new ``ip`` address value, ``constants.NIC_IP_POOL``, |
122 |
that specifies that a given NIC's IP address should be obtained using |
123 |
the IP address pool of the specified network. This value is only valid |
124 |
for managed-mode NICs, where it is also used as a default instead of |
125 |
``constants.VALUE_AUTO``. A managed-mode NIC's IP address can also be |
126 |
specified manually, as long as it is compatible with the network the NIC |
127 |
is connected to. |
128 |
|
129 |
|
130 |
Hooks |
131 |
+++++ |
132 |
|
133 |
``OP_NETWORK_ADD`` |
134 |
Add a network to Ganeti |
135 |
|
136 |
:directory: network-add |
137 |
:pre-execution: master node |
138 |
:post-execution: master node |
139 |
|
140 |
``OP_NETWORK_CONNECT`` |
141 |
Connect a network to a node group. This hook can be used to e.g. |
142 |
configure network interfaces on the group's nodes. |
143 |
|
144 |
:directory: network-connect |
145 |
:pre-execution: master node, all nodes in the connected group |
146 |
:post-execution: master node, all nodes in the connected group |
147 |
|
148 |
``OP_NETWORK_DISCONNECT`` |
149 |
Disconnect a network to a node group. This hook can be used to e.g. |
150 |
deconfigure network interfaces on the group's nodes. |
151 |
|
152 |
:directory: network-disconnect |
153 |
:pre-execution: master node, all nodes in the connected group |
154 |
:post-execution: master node, all nodes in the connected group |
155 |
|
156 |
``OP_NETWORK_REMOVE`` |
157 |
Remove a network from Ganeti |
158 |
|
159 |
:directory: network-add |
160 |
:pre-execution: master node, all nodes |
161 |
:post-execution: master node, all nodes |
162 |
|
163 |
Hook variables |
164 |
^^^^^^^^^^^^^^ |
165 |
|
166 |
``INSTANCE_NICn_MANAGED`` |
167 |
Non-zero if NIC n is a managed-mode NIC |
168 |
|
169 |
``INSTANCE_NICn_NETWORK`` |
170 |
The friendly name of the network |
171 |
|
172 |
``INSTANCE_NICn_NETWORK_UUID`` |
173 |
The network's UUID |
174 |
|
175 |
``INSTANCE_NICn_NETWORK_TAGS`` |
176 |
The network's tags |
177 |
|
178 |
``INSTANCE_NICn_NETWORK_IPV4_CIDR``, ``INSTANCE_NICn_NETWORK_IPV6_CIDR`` |
179 |
The subnet in CIDR notation |
180 |
|
181 |
``INSTANCE_NICn_NETWORK_IPV4_GATEWAY``, ``INSTANCE_NICn_NETWORK_IPV6_GATEWAY`` |
182 |
The subnet's default gateway |
183 |
|
184 |
|
185 |
Backend changes |
186 |
+++++++++++++++ |
187 |
|
188 |
In order to keep the hypervisor-visible changes to a minimum, and |
189 |
maintain compatibility with the existing network configuration scripts, |
190 |
the instance's hypervisor configuration will have host-level link and |
191 |
mode replaced by the *connectivity mode* and *host interface* of the |
192 |
given network on the current node group. |
193 |
|
194 |
The managed mode can be detected by the presence of new environment |
195 |
variables in network configuration scripts: |
196 |
|
197 |
Network configuration script variables |
198 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
199 |
|
200 |
``MANAGED`` |
201 |
Non-zero if NIC is a managed-mode NIC |
202 |
|
203 |
``NETWORK`` |
204 |
The friendly name of the network |
205 |
|
206 |
``NETWORK_UUID`` |
207 |
The network's UUID |
208 |
|
209 |
``NETWORK_TAGS`` |
210 |
The network's tags |
211 |
|
212 |
``NETWORK_IPv4_CIDR``, ``NETWORK_IPv6_CIDR`` |
213 |
The subnet in CIDR notation |
214 |
|
215 |
``NETWORK_IPV4_GATEWAY``, ``NETWORK_IPV6_GATEWAY`` |
216 |
The subnet's default gateway |
217 |
|
218 |
Userland interface |
219 |
++++++++++++++++++ |
220 |
|
221 |
A new client script is introduced, ``gnt-network``, which handles |
222 |
network-related configuration in Ganeti. |
223 |
|
224 |
Network addition/deletion |
225 |
^^^^^^^^^^^^^^^^^^^^^^^^^ |
226 |
:: |
227 |
|
228 |
gnt-network add --cidr=192.0.2.0/24 --gateway=192.0.2.1 \ |
229 |
--cidr6=2001:db8:2ffc::/64 --gateway6=2001:db8:2ffc::1 \ |
230 |
--nic_connectivity=bridged --host_interface=br0 public |
231 |
gnt-network remove public (only allowed if no instances are using the network) |
232 |
|
233 |
Manual IP address reservation |
234 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
235 |
:: |
236 |
|
237 |
gnt-network reserve-ips public 192.0.2.2 192.0.2.10-192.0.2.20 |
238 |
gnt-network release-ips public 192.0.2.3 |
239 |
|
240 |
|
241 |
Network modification |
242 |
^^^^^^^^^^^^^^^^^^^^ |
243 |
:: |
244 |
|
245 |
gnt-network modify --cidr=192.0.2.0/25 public (only allowed if all current reservations fit in the new network) |
246 |
gnt-network modify --gateway=192.0.2.126 public |
247 |
gnt-network modify --host_interface=test --nic_connectivity=routed public (issues warning about instances that need to be rebooted) |
248 |
gnt-network rename public public2 |
249 |
|
250 |
|
251 |
Assignment to node groups |
252 |
^^^^^^^^^^^^^^^^^^^^^^^^^ |
253 |
:: |
254 |
|
255 |
gnt-network connect public nodegroup1 |
256 |
gnt-network connect --host_interface=br1 public nodegroup2 |
257 |
gnt-network disconnect public nodegroup1 (only permitted if no instances are currently using this network in the group) |
258 |
|
259 |
Tagging |
260 |
^^^^^^^ |
261 |
:: |
262 |
|
263 |
gnt-network add-tags public foo bar:baz |
264 |
|
265 |
Network listing |
266 |
^^^^^^^^^^^^^^^ |
267 |
:: |
268 |
|
269 |
gnt-network list |
270 |
Name IPv4 Network IPv4 Gateway IPv6 Network IPv6 Gateway Connected to |
271 |
public 192.0.2.0/24 192.0.2.1 2001:db8:dead:beef::/64 2001:db8:dead:beef::1 nodegroup1:br0 |
272 |
private 10.0.1.0/24 - - - |
273 |
|
274 |
Network information |
275 |
^^^^^^^^^^^^^^^^^^^ |
276 |
:: |
277 |
|
278 |
gnt-network info public |
279 |
Name: public |
280 |
IPv4 Network: 192.0.2.0/24 |
281 |
IPv4 Gateway: 192.0.2.1 |
282 |
IPv6 Network: 2001:db8:dead:beef::/64 |
283 |
IPv6 Gateway: 2001:db8:dead:beef::1 |
284 |
Total IPv4 count: 256 |
285 |
Free address count: 201 (80% free) |
286 |
IPv4 pool status: XXX.........XXXXXXXXXXXXXX...XX............. |
287 |
XXX..........XXX...........................X |
288 |
....XXX..........XXX.....................XXX |
289 |
X: occupied .: free |
290 |
Externally reserved IPv4 addresses: |
291 |
192.0.2.3, 192.0.2.22 |
292 |
Connected to node groups: |
293 |
default (link br0), other_group(link br1) |
294 |
Used by 22 instances: |
295 |
inst1 |
296 |
inst2 |
297 |
inst32 |
298 |
.. |
299 |
|
300 |
|
301 |
IAllocator changes |
302 |
++++++++++++++++++ |
303 |
|
304 |
The IAllocator protocol can be made network-aware, i.e. also consider |
305 |
network availability for node group selection. Networks, as well as |
306 |
future shared storage pools, can be seen as constraints used to rule out |
307 |
the placement on certain node groups. |
308 |
|
309 |
.. vim: set textwidth=72 : |
310 |
.. Local Variables: |
311 |
.. mode: rst |
312 |
.. fill-column: 72 |
313 |
.. End: |