5 .. contents:: :depth: 4
7 This is a design document detailing the implementation of network resource
10 Current state and shortcomings
11 ==============================
13 Currently Ganeti supports two configuration modes for instance NICs:
14 routed and bridged mode. The ``ip`` NIC parameter, which is mandatory
15 for routed NICs and optional for bridged ones, holds the given NIC's IP
16 address and may be filled either manually, or via a DNS lookup for the
19 This approach presents some shortcomings:
21 a) It relies on external systems to perform network resource
22 management. Although large organizations may already have IP pool
23 management software in place, this is not usually the case with
24 stand-alone deployments. For smaller installations it makes sense to
25 allocate a pool of IP addresses to Ganeti and let it transparently
26 assign these IPs to instances as appropriate.
28 b) The NIC network information is incomplete, lacking netmask and
29 gateway. Operating system providers could for example use the
30 complete network information to fully configure an instance's
31 network parameters upon its creation.
33 Furthermore, having full network configuration information would
34 enable Ganeti nodes to become more self-contained and be able to
35 infer system configuration (e.g. /etc/network/interfaces content)
36 from Ganeti configuration. This should make configuration of
37 newly-added nodes a lot easier and less dependant on external
40 c) Instance placement must explicitly take network availability in
41 different node groups into account; the same ``link`` is implicitly
42 expected to connect to the same network across the whole cluster,
43 which may not always be the case with large clusters with multiple
50 In order to deal with the above shortcomings, we propose to extend
51 Ganeti with high-level network management logic, which consists of a new
52 NIC mode called ``managed``, a new "Network" configuration object and
53 logic to perform IP address pool management, i.e. maintain a set of
54 available and occupied IP addresses.
59 We propose the introduction of a new high-level Network object,
60 containing (at least) the following data:
64 - Network in CIDR notation (IPv4 + IPv6)
65 - Default gateway, if one exists (IPv4 + IPv6)
66 - IP pool management data (reservations)
67 - Default NIC connectivity mode (bridged, routed). This is the
68 functional equivalent of the current NIC ``mode``.
69 - Default host interface (e.g. br0). This is the functional equivalent
70 of the current NIC ``link``.
73 Each network will be connected to any number of node groups, possibly
74 overriding connectivity mode and host interface for each node group.
75 This is achieved by adding a ``networks`` slot to the NodeGroup object
76 and using the networks' UUIDs as keys.
81 A new helper library is introduced, wrapping around Network objects to
82 give IP pool management capabilities. A network's pool is defined by two
83 bitfields, the length of the network size each:
86 This field holds all IP addresses reserved by Ganeti instances, as
87 well as cluster IP addresses (node addresses + cluster master)
89 ``external reservations``
90 This field holds all IP addresses that are manually reserved by the
91 administrator, because some other equipment is using them outside the
94 The bitfields are implemented using the python-bitarray package for
95 space efficiency and their binary value stored base64-encoded for JSON
96 compatibility. This approach gives relatively compact representations
97 even for large IPv4 networks (e.g. /20).
99 Ganeti-owned IP addresses (node + master IPs) are reserved automatically
100 if the cluster's data network itself is placed under pool management.
102 Helper ConfigWriter methods provide free IP address generation and
103 reservation, using a TemporaryReservationManager.
105 It should be noted that IP pool management is performed only for IPv4
106 networks, as they are expected to be densely populated. IPv6 networks
107 can use different approaches, e.g. sequential address asignment or
113 In order to be able to use the new network facility while maintaining
114 compatibility with the current networking model, a new network mode is
115 introduced, called ``managed`` to reflect the fact that the given NICs
116 network configuration is managed by Ganeti itself. A managed mode NIC
117 accepts the network it is connected to in its ``link`` argument.
118 Userspace tools can refer to networks using their symbolic names,
119 however internally, the link argument stores the network's UUID.
121 We also introduce a new ``ip`` address value, ``constants.NIC_IP_POOL``,
122 that specifies that a given NIC's IP address should be obtained using
123 the IP address pool of the specified network. This value is only valid
124 for managed-mode NICs, where it is also used as a default instead of
125 ``constants.VALUE_AUTO``. A managed-mode NIC's IP address can also be
126 specified manually, as long as it is compatible with the network the NIC
134 Add a network to Ganeti
136 :directory: network-add
137 :pre-execution: master node
138 :post-execution: master node
140 ``OP_NETWORK_CONNECT``
141 Connect a network to a node group. This hook can be used to e.g.
142 configure network interfaces on the group's nodes.
144 :directory: network-connect
145 :pre-execution: master node, all nodes in the connected group
146 :post-execution: master node, all nodes in the connected group
148 ``OP_NETWORK_DISCONNECT``
149 Disconnect a network to a node group. This hook can be used to e.g.
150 deconfigure network interfaces on the group's nodes.
152 :directory: network-disconnect
153 :pre-execution: master node, all nodes in the connected group
154 :post-execution: master node, all nodes in the connected group
156 ``OP_NETWORK_REMOVE``
157 Remove a network from Ganeti
159 :directory: network-add
160 :pre-execution: master node, all nodes
161 :post-execution: master node, all nodes
166 ``INSTANCE_NICn_MANAGED``
167 Non-zero if NIC n is a managed-mode NIC
169 ``INSTANCE_NICn_NETWORK``
170 The friendly name of the network
172 ``INSTANCE_NICn_NETWORK_UUID``
175 ``INSTANCE_NICn_NETWORK_TAGS``
178 ``INSTANCE_NICn_NETWORK_IPV4_CIDR``, ``INSTANCE_NICn_NETWORK_IPV6_CIDR``
179 The subnet in CIDR notation
181 ``INSTANCE_NICn_NETWORK_IPV4_GATEWAY``, ``INSTANCE_NICn_NETWORK_IPV6_GATEWAY``
182 The subnet's default gateway
188 In order to keep the hypervisor-visible changes to a minimum, and
189 maintain compatibility with the existing network configuration scripts,
190 the instance's hypervisor configuration will have host-level link and
191 mode replaced by the *connectivity mode* and *host interface* of the
192 given network on the current node group.
194 The managed mode can be detected by the presence of new environment
195 variables in network configuration scripts:
197 Network configuration script variables
198 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
201 Non-zero if NIC is a managed-mode NIC
204 The friendly name of the network
212 ``NETWORK_IPv4_CIDR``, ``NETWORK_IPv6_CIDR``
213 The subnet in CIDR notation
215 ``NETWORK_IPV4_GATEWAY``, ``NETWORK_IPV6_GATEWAY``
216 The subnet's default gateway
221 A new client script is introduced, ``gnt-network``, which handles
222 network-related configuration in Ganeti.
224 Network addition/deletion
225 ^^^^^^^^^^^^^^^^^^^^^^^^^
228 gnt-network add --cidr=192.0.2.0/24 --gateway=192.0.2.1 \
229 --cidr6=2001:db8:2ffc::/64 --gateway6=2001:db8:2ffc::1 \
230 --nic_connectivity=bridged --host_interface=br0 public
231 gnt-network remove public (only allowed if no instances are using the network)
233 Manual IP address reservation
234 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
237 gnt-network reserve-ips public 192.0.2.2 192.0.2.10-192.0.2.20
238 gnt-network release-ips public 192.0.2.3
245 gnt-network modify --cidr=192.0.2.0/25 public (only allowed if all current reservations fit in the new network)
246 gnt-network modify --gateway=192.0.2.126 public
247 gnt-network modify --host_interface=test --nic_connectivity=routed public (issues warning about instances that need to be rebooted)
248 gnt-network rename public public2
251 Assignment to node groups
252 ^^^^^^^^^^^^^^^^^^^^^^^^^
255 gnt-network connect public nodegroup1
256 gnt-network connect --host_interface=br1 public nodegroup2
257 gnt-network disconnect public nodegroup1 (only permitted if no instances are currently using this network in the group)
263 gnt-network add-tags public foo bar:baz
270 Name IPv4 Network IPv4 Gateway IPv6 Network IPv6 Gateway Connected to
271 public 192.0.2.0/24 192.0.2.1 2001:db8:dead:beef::/64 2001:db8:dead:beef::1 nodegroup1:br0
272 private 10.0.1.0/24 - - -
278 gnt-network info public
280 IPv4 Network: 192.0.2.0/24
281 IPv4 Gateway: 192.0.2.1
282 IPv6 Network: 2001:db8:dead:beef::/64
283 IPv6 Gateway: 2001:db8:dead:beef::1
284 Total IPv4 count: 256
285 Free address count: 201 (80% free)
286 IPv4 pool status: XXX.........XXXXXXXXXXXXXX...XX.............
287 XXX..........XXX...........................X
288 ....XXX..........XXX.....................XXX
290 Externally reserved IPv4 addresses:
291 192.0.2.3, 192.0.2.22
292 Connected to node groups:
293 default (link br0), other_group(link br1)
294 Used by 22 instances:
304 The IAllocator protocol can be made network-aware, i.e. also consider
305 network availability for node group selection. Networks, as well as
306 future shared storage pools, can be seen as constraints used to rule out
307 the placement on certain node groups.
309 .. vim: set textwidth=72 :