root / docs / networks.rst @ 210e5933
History | View | Annotate | Download (18.2 kB)
1 |
.. _networks: |
---|---|
2 |
|
3 |
Network Service (part of Cyclades) |
4 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
5 |
|
6 |
Network setup overview |
7 |
====================== |
8 |
|
9 |
Currently the Network Service is part of Cyclades and comes along with the |
10 |
Cyclades software components. |
11 |
|
12 |
Networking is deployment-specific and must be customized based on the specific |
13 |
needs of the system administrator. However, to do so, the administrator needs |
14 |
to understand how each level handles Virtual Networks, to be able to setup the |
15 |
backend appropriately. |
16 |
|
17 |
Since v0.11 Synnefo supports multiple Ganeti clusters (backends). Having in |
18 |
mind that every backend has its locality, there is a high possibility each |
19 |
cluster to have different infrastracture (wires, routers, subnets, gateways, |
20 |
etc.). |
21 |
|
22 |
In the following sections we investigate in a top-down approach, the way |
23 |
networks are defined from the Cyclades, Ganeti, and Backend persperctive. |
24 |
|
25 |
Network @ Cyclades level |
26 |
------------------------ |
27 |
|
28 |
Cyclades networks support a range of different options to cover the specific |
29 |
needs of each deployment. |
30 |
|
31 |
First of all, as far as visibility and accessibility is concerned, a network |
32 |
can be either `public` or `private`. Public networks are created by the |
33 |
administrator via the command line interface (`snf-manage`) and are visible to |
34 |
all end-users. On the other hand, private networks are created by the end-user |
35 |
from the Web UI or the kamaki client and provide isolated Layer 2 connectivity |
36 |
to the end-user. |
37 |
|
38 |
Both networks can have an IPv4 subnet or/and an IPv6 subnet along with the |
39 |
corresponding gateway. For IPv4 networks, if the `--dhcp` option is set, |
40 |
Cyclades will treat the IPv4 subnet as an IP pool, and will assign to each VM |
41 |
that is connected to this network an IPv4 address from this pool. |
42 |
|
43 |
A public network can also be marked as a floating IP pool with the |
44 |
`--floating-ip-pool` option. Floating IPs, are IPv4 addresses that can be |
45 |
dynamically by added and removed from running VMs. A user can reserve and |
46 |
release a floating IP address that he can later add and remove it from running |
47 |
VMs. Also the user can release a floating IP if it not used by any of his |
48 |
VMs. |
49 |
|
50 |
Private networks and floating IPs must be accessible from all instances across |
51 |
all Ganeti backends. So, such networks must exist in all backends, and |
52 |
are dynamically created when new Ganeti backends are added. Specially for |
53 |
private networks, to avoid the overhead of creating the network to all |
54 |
backends, Cyclades create these networks on demand, when an instance that |
55 |
lives in a backend tries to connect to this network. |
56 |
|
57 |
The administrator may also want to connect instances to some network, without |
58 |
supporting floating IPs (e.g. to enforce each VM to be connected to a specific |
59 |
network). This can be achieved by setting the `DEFAULT_INSTANCE_NETWORKS` |
60 |
setting to the list of the selected networks. The special keyword |
61 |
`SNF:ANY_PUBLIC` may be used as a network identifier, to indicate to the system |
62 |
to peak any of the public networks that has a free IP address. Public networks |
63 |
that are not floating IP pools, do not need to exist to all Ganeti backends, |
64 |
since the Cyclades backend allocator, will route spawned vms to a Ganeti |
65 |
backend that the selected networks exist. The administrator can choose in |
66 |
which backends to create the network via the `--backends` command line option. |
67 |
|
68 |
Another distinction between networks is their flavor. Flavor is a way to |
69 |
abstract infrastructure specific options, that are used to ensure connectivity |
70 |
and isolation to the VMs connected to the network. It is a set of options that |
71 |
eventually will guide scripts to set up rules, while creating virtual |
72 |
interfaces in the node level. Each of these flavors define attributes that will |
73 |
be used at Ganeti level to create the physical network. These attributes are: |
74 |
|
75 |
* ``mode``: Whether the network is in 'bridged' or 'routed' mode. |
76 |
* ``link``: Bridge for 'bridged' networks and routing table for 'routed' |
77 |
networks. e.g. 'br100', 'rt200' |
78 |
* ``mac_prefix``: A MAC prefix for the network. e.g. 'aa:00:05' |
79 |
* ``tags``: A list of tags to be used at the Ganeti level. |
80 |
|
81 |
To ensure L2 isolation, Synnefo supports two different mechanisms (see also Node |
82 |
Level section): |
83 |
|
84 |
* assigning one physical VLAN per network |
85 |
* assigning one MAC prefix per network, so that every NIC attached to this |
86 |
network will have this prefix. Isolation is then achieved by filtering |
87 |
rules (via `ebtables`) based on a specific mask (ff:ff:f0:00:00:00, see Node |
88 |
Level section for more details). |
89 |
|
90 |
Having this in mind and in order to prevent assignment of duplicate VLAN/MAC |
91 |
prefix to different networks, Synnefo supports two types of Pools: |
92 |
|
93 |
- Bridge Pool (corresponding to a number of VLANs bridged to those bridges) |
94 |
- MAC prefix Pool |
95 |
|
96 |
For Pool handling refer to the corresponding doc section. To use this pools, |
97 |
set either `--link` or `--mac-prefix` to the reserved keyword `pool`. |
98 |
|
99 |
Existing network flavors are the following: |
100 |
|
101 |
============== ======= =============================== ====================== ================== |
102 |
Flavor Name Mode Link MAC prefix Tags |
103 |
============== ======= =============================== ====================== ================== |
104 |
IP_LESS_ROUTED routed ``DEFAULT_ROUTING_TABLE`` ``DEFAULT_MAC_PREFIX`` 'ip-less-routed' |
105 |
MAC_FILTERED bridged ``DEFAULT_MAC_FILTERED_BRIDGE`` 'pool' 'private'filtered' |
106 |
PHYSICAL_VLAN bridged 'pool' ``DEFAULT_MAC_PREFIX`` 'physical-vlan' |
107 |
CUSTOM bridged ``DEFAULT_BRIDGE`` ``DEFAULT_MAC_PREFIX`` |
108 |
============== ======= =============================== ====================== ================== |
109 |
|
110 |
``DEFAULT_ROUTING_TABLE``, ``DEFAULT_MAC_PREFIX``, ``DEFAULT_BRIDGE``, |
111 |
``DEFAULT_MAC_FILTERED_BRIDGE`` are all configurable settings in |
112 |
``/etc/synnefo/20-snf-cyclades-app-api.conf``. 'pool' is used to denote that a |
113 |
link or MAC prefix will be allocated from the corresponding Pool. Finally, |
114 |
most of these attributes, may be overridden when creating networks with |
115 |
`snf-manage network-create command`. |
116 |
|
117 |
The administrator is able to create any of the above flavors |
118 |
and override their default values by explicitly passing mode, link, etc. using |
119 |
the `snf-manage network-create` command. |
120 |
|
121 |
The administrator can create networks of any flavor, but end-users is allowed |
122 |
to create via API only networks with flavors that are set in the |
123 |
`API_ENABLED_NETWORK_FLAVORS` setting. |
124 |
|
125 |
Network @ Ganeti level |
126 |
---------------------- |
127 |
|
128 |
Currently, stable Ganeti does not support IP Pool management. However, the |
129 |
functionality has been merged in the official Ganeti master branch and will |
130 |
appear on Ganeti 2.7.0. So, you can either checkout the Ganeti master branch |
131 |
and build your packages, or clone our local repo |
132 |
https://code.grnet.gr/git/ganeti-local and checkout the |
133 |
`stable-2.6-ippool-hotplug-esi` branch. This is the Ganeti stable branch with |
134 |
IP pool management, Hotplugging and ExtStorage Interface features merged on top |
135 |
of it. The last two features are not a hard Synnefo requirement, but will |
136 |
enable you to do neat things when you get experienced with Synnefo. They are |
137 |
going to be pushed for review upstream sometime soon. |
138 |
|
139 |
Any network created in Synnefo is also created in one (for public networks) or |
140 |
all (for private networks) Ganeti backends. In Ganeti a network can have the |
141 |
following options: |
142 |
|
143 |
- network (192.168.0.0/24, mandatory) |
144 |
- gateway (192.168.0.1) |
145 |
- network6 (2001:648:2ffc:1201::/64) |
146 |
- gateway6 (2001:648:2ffc:1201::1) |
147 |
- mac_prefix (aa:00:01) |
148 |
- type (private, public) |
149 |
- tags |
150 |
|
151 |
Networks in Ganeti cannot be used unless they are connected to a nodegroup in |
152 |
order to define the connectivity mode and link. Synnefo, after creating a |
153 |
network, connects it to all nodegroups of the Ganeti cluster(s) with the given |
154 |
mode and link (defined in the network flavor). |
155 |
|
156 |
Ganeti makes use of environment variables to inform scripts about each NIC's |
157 |
setup. `kvm-vif-script` that comes with `snf-network` sets up the nfdhcpd lease and |
158 |
applies any rules needed depending on the network's mode, link, mac_prefix and |
159 |
tags. |
160 |
|
161 |
Network @ Physical host level |
162 |
----------------------------- |
163 |
|
164 |
Currently, networking infrastructure must be pre-provisioned before creating |
165 |
networks in Synnefo. According to which flavors you want to support, you should |
166 |
have already setup all your physical hosts correspondingly. This means you |
167 |
need: |
168 |
|
169 |
- one bridge for the ``CUSTOM`` flavor (br0, see Fig. 1) |
170 |
- one bridge for the ``MAC_FILTERED`` flavor (prv0, see Fig. 2) |
171 |
- a number of bridges and their corresponding VLANs (bridged to them) for |
172 |
the ``PHYSICAL_VLAN`` flavor (prv1..prv100, see Fig. 3) |
173 |
- a routing table for the ``IP_LESS_ROUTED`` flavor (snf_public, see Fig. 4) |
174 |
|
175 |
Please refer to the following figures, which clarify each infrastructure setup |
176 |
and how connectivity and isolation is achieved in every case for every type of |
177 |
network. |
178 |
|
179 |
|
180 |
FLAVORS |
181 |
======= |
182 |
|
183 |
As mentioned earlier supported flavors are: |
184 |
|
185 |
- CUSTOM |
186 |
- IP_LESS_ROUTED |
187 |
- MAC_FILTERED |
188 |
- PHYSICAL_VLAN |
189 |
|
190 |
In the following sections we mention what configuration imposes each flavor from |
191 |
Synnefo, Ganeti and Physical host perspective. |
192 |
|
193 |
|
194 |
|
195 |
DEFAULT SCENARIO |
196 |
---------------- |
197 |
|
198 |
In this case we will bridge all primary interfaces of the VMs on one bridge that must |
199 |
be the same collition domain with the router. The router sould then forward packets |
200 |
(if a public IPv4 Subnet is available) or do NAT in order to provide internet access to |
201 |
the VMs. |
202 |
|
203 |
To this end we will use the CUSTOM flavor and pre-provision in each Ganeti |
204 |
node one bridge (e.g. ``br100``). If we assume that ``eth1`` is the physical interface |
205 |
connected to the router, run: |
206 |
|
207 |
.. image:: images/network-bridged.png |
208 |
:align: right |
209 |
:height: 550px |
210 |
:width: 500px |
211 |
|
212 |
.. code-block:: console |
213 |
|
214 |
# brctl addbr br100 |
215 |
# brctl addif br100 eth1 |
216 |
# ip link set br100 up |
217 |
|
218 |
# brctl show |
219 |
bridge name bridge id STP enabled interfaces |
220 |
br100 8000.8a3c3ede3583 no eth1 |
221 |
|
222 |
|
223 |
|
224 |
Then in Cyclades run: |
225 |
|
226 |
.. code-block:: console |
227 |
|
228 |
# snf-manage network-create --subnet=5.6.7.0/27 --gateway=5.6.7.1 --subnet6=2001:648:2FFC:1322::/64 --gateway6=2001:648:2FFC:1322::1 --public --dhcp=True --flavor=CUSTOM --link=br100 ----name=default --backend-id=1 |
229 |
|
230 |
# snf-manage network-list |
231 |
id name flavor owner mac_prefix dhcp state link vms public IPv4 Subnet IPv4 Gateway |
232 |
1 default CUSTOM True ACTIVE br100 True 5.6.7.0/27 5.6.7.1 |
233 |
|
234 |
This will add a network in Synnefo DB and create a network in Ganeti backend by |
235 |
issuing: |
236 |
|
237 |
.. code-block:: console |
238 |
|
239 |
# gnt-network add --network=5.6.7.0/27 --gateway=5.6.7.1 --network6=2001:648:2FFC:1322::/64 --gateway6=2001:648:2FFC:1322::1 --network-type=public --tags=nfdhcpd snf-net-1 |
240 |
|
241 |
# gnt-network connect snf-net-1 default bridged br100 |
242 |
# gnt-network list snf-net-1 |
243 |
Network Subnet Gateway NetworkType MacPrefix GroupList Tags |
244 |
snf-net-1 5.6.7.0/27 5.6.7.1 public None default(bridged, br100) nfdhcpd |
245 |
|
246 |
|
247 |
To enable NAT in a Internal Router if you do not have a public IP range available |
248 |
but only a public routable IP (e.g 1.2.3.4): |
249 |
|
250 |
.. code-block:: console |
251 |
|
252 |
# ip addr add 5.6.7.1/27 dev eth1 |
253 |
# iptables -t nat -A POSTROUTING -o eth1 --to-source 1.2.3.4 -j SNAT |
254 |
|
255 |
IP_LESS_ROUTED |
256 |
-------------- |
257 |
|
258 |
.. image:: images/network-routed.png |
259 |
:align: right |
260 |
:height: 580px |
261 |
:width: 500px |
262 |
|
263 |
To create a network with IP_LESS_ROUTED flavor run you have to pre-provision in |
264 |
each Ganeti node one routing table (e.g. ``snf_public``) that will do all the |
265 |
routing from/to the VMs' taps. Additionally you must enable ``Proxy-ARP`` |
266 |
support. All traffic will be on a single iterface (e.g. ``eth1``). |
267 |
|
268 |
.. code-block:: console |
269 |
|
270 |
# echo 1 > /proc/sys/net/ipv4/conf/ip_fowarding |
271 |
# echo 10 snf_public >> /etc/iproute2/rt_tables |
272 |
# ip route add 5.6.7.0/27 dev eth1 |
273 |
# ip route add 5.6.7.0/27 dev eth1 table snf_public |
274 |
# ip route add default via 5.6.7.1 dev eth1 table snf_public |
275 |
# ip rule add iif eth1 lookup snf_public |
276 |
# arptables -A OUTPUT -o eth1 --opcode 1 --mangle-ip-s 5.6.7.30 # last ip in Subnet |
277 |
|
278 |
Then in Cyclades run: |
279 |
|
280 |
.. code-block:: console |
281 |
|
282 |
# snf-manage network-create --subnet=5.6.7.0/27 --gateway=5.6.7.1 --subnet6=2001:648:2FFC:1322::/64 --gateway6=2001:648:2FFC:1322::1 --public --dhcp=True --flavor=IP_LESS_ROUTED --name=routed --backend-id=1 |
283 |
|
284 |
# snf-manage network-list |
285 |
id name flavor owner mac_prefix dhcp state link vms public IPv4 Subnet IPv4 Gateway |
286 |
2 routed IP_LESS_ROUTED True ACTIVE snf_public True 5.6.7.0/27 5.6.7.1 |
287 |
|
288 |
|
289 |
This will add a network in Synnefo DB and create a network in Ganeti backend by |
290 |
issuing: |
291 |
|
292 |
.. code-block:: console |
293 |
|
294 |
# gnt-network add --network=5.6.7.0/27 --gateway=5.6.7.1 --network6=2001:648:2FFC:1322::/64 --gateway6=2001:648:2FFC:1322::1 --network-type=public --tags=nfdhcpd,ip-less-routed snf-net-2 |
295 |
|
296 |
# gnt-network connect snf-net-2 default bridged br100 |
297 |
# gnt-network list snf-net-2 |
298 |
Network Subnet Gateway NetworkType MacPrefix GroupList Tags |
299 |
dimara-net-1 62.217.123.128/27 62.217.123.129 public None default(routed, snf_public) nfdhcpd,ip-less-routed |
300 |
|
301 |
|
302 |
|
303 |
|
304 |
MAC_FILTERED |
305 |
------------ |
306 |
|
307 |
|
308 |
To create a network with MAC_FILTERED flavor you have to pre-provision in each Ganeti |
309 |
node one bridge (e.g. ``prv0``) that will be bridged with one interface (e.g. ``eth2``) |
310 |
across the whole cluster. |
311 |
|
312 |
.. image:: images/network-mac.png |
313 |
:align: right |
314 |
:height: 500px |
315 |
:width: 500px |
316 |
|
317 |
.. code-block:: console |
318 |
|
319 |
# brctl addbr prv0 |
320 |
# brctl addif prv0 eth2 |
321 |
# ip link set prv0 up |
322 |
|
323 |
# brctl show |
324 |
bridge name bridge id STP enabled interfaces |
325 |
prv0 8000.8a3c3ede3583 no eth2 |
326 |
|
327 |
|
328 |
|
329 |
Then in Cyclades first create a pool for MAC prefixes by running: |
330 |
|
331 |
.. code-block:: console |
332 |
|
333 |
# snf-manage pool-create --type=mac-prefix --base=aa:00:0 --size=65536 |
334 |
|
335 |
and the create the network: |
336 |
|
337 |
.. code-block:: console |
338 |
|
339 |
# snf-manage network-create --subnet=192.168.1.0/24 --gateway=192.168.1.0/24 --dhcp=True --flavor=MAC_FILTERED --link=prv0 --name=mac --backend-id=1 |
340 |
# snf-manage network-list |
341 |
id name flavor owner mac_prefix dhcp state link vms public IPv4 Subnet IPv4 Gateway |
342 |
3 mac MAC_FILTERED aa:00:01 True ACTIVE prv0 False 192.168.1.0/24 192.168.1.1 |
343 |
|
344 |
Edit the synnefo setting `DEFAULT_MAC_FILTERED_BRIDGE` to `prv0`. |
345 |
|
346 |
This will add a network in Synnefo DB and create a network in Ganeti backend by |
347 |
issuing: |
348 |
|
349 |
.. code-block:: console |
350 |
|
351 |
# gnt-network add --network=192.168.1.0/24 --gateway=192.168.1.1 --network-type=private --tags=nfdhcpd,private-filtered snf-net-3 |
352 |
|
353 |
# gnt-network connect snf-net-3 default bridged prv0 |
354 |
# gnt-network list snf-net-3 |
355 |
Network Subnet Gateway NetworkType MacPrefix GroupList Tags |
356 |
snf-net-3 192.168.1.0/24 192.168.1.1 private aa:00:01 default(bridged, prv0) nfdhcpd,private-filtered |
357 |
|
358 |
|
359 |
|
360 |
|
361 |
|
362 |
|
363 |
PHYSICAL_VLAN |
364 |
------------- |
365 |
|
366 |
|
367 |
To create a network with PHYSICAL_VALN flavor you have to pre-provision in each Ganeti |
368 |
node a range of bridges (e.g. ``prv1..20``) that will be bridged with the corresponding VLANs (e.g. ``401..420``) |
369 |
across the whole cluster. To this end if we assume that ``eth3`` is the interface to use, run: |
370 |
|
371 |
.. image:: images/network-vlan.png |
372 |
:align: right |
373 |
:height: 480px |
374 |
:width: 500px |
375 |
|
376 |
|
377 |
.. code-block:: console |
378 |
|
379 |
# for i in {1..20}; do |
380 |
br=prv$i ; vlanid=$((400+i)) ; vlan=eth3.$vlanid |
381 |
brctl addbr $br ; ip link set $br up |
382 |
vconfig add eth0 vlanid ; ip link set vlan up |
383 |
brctl addif $br $vlan |
384 |
done |
385 |
# brctl show |
386 |
bridge name bridge id STP enabled interfaces |
387 |
prv1 8000.8a3c3ede3583 no eth3.401 |
388 |
prv2 8000.8a3c3ede3583 no eth3.402 |
389 |
... |
390 |
|
391 |
|
392 |
Then in Cyclades first create a pool for bridges by running: |
393 |
|
394 |
.. code-block:: console |
395 |
|
396 |
# snf-manage pool-create --type=bridge --base=prv --size=20 |
397 |
|
398 |
and the create the network: |
399 |
|
400 |
.. code-block:: console |
401 |
|
402 |
# snf-manage network-create --subnet=192.168.1.0/24 --gateway=192.168.1.0/24 --dhcp=True --flavor=PHYSICAL_VLAN --name=vlan --backend-id=1 |
403 |
|
404 |
# snf-manage network-list |
405 |
id name flavor owner mac_prefix dhcp state link vms public IPv4 Subnet IPv4 Gateway |
406 |
4 vlan PHYSICAL_VLAN True ACTIVE prv1 False 192.168.1.0/24 192.168.1.1 |
407 |
|
408 |
This will add a network in Synnefo DB and create a network in Ganeti backend by |
409 |
issuing: |
410 |
|
411 |
.. code-block:: console |
412 |
|
413 |
# gnt-network add --network=192.168.1.0/24 --gateway=192.168.1.1 --network-type=private --tags=nfdhcpd,physica-vlan snf-net-4 |
414 |
|
415 |
# gnt-network connect snf-net-4 default bridged prv1 |
416 |
# gnt-network list snf-net-4 |
417 |
Network Subnet Gateway NetworkType MacPrefix GroupList Tags |
418 |
snf-net-4 192.168.1.0/24 192.168.1.1 private None default(bridged, prv1) nfdhcpd,physical-vlan |
419 |
|
420 |
|
421 |
|
422 |
ADVANCED SCENARIO |
423 |
----------------- |
424 |
|
425 |
To create a network with CUSTOM flavor you have to pass your self mode, link, |
426 |
mac prefix, tags for the network. You are not allowed to use the existing pools |
427 |
(only MAC_FILTERED, PHYSICAL_VLAN use them) so link and mac prefix uniqueness |
428 |
cannot be guaranteed. |
429 |
|
430 |
Lets assume a bridge ``br200`` that serves a VPN network to GRNET already exists |
431 |
on Ganeti nodes and we want to create for a certain user a private network so |
432 |
that he can access the VPN. Then we run in Cyclades: |
433 |
|
434 |
.. code-block:: console |
435 |
|
436 |
# snf-manage network-create --subnet=192.168.1.0/24 --gateway=192.168.1.0/24 --dhcp=True --flavor=CUSTOM --mode=bridged --link=br200 --mac-prefix=bb:00:44 --owner=user@grnet.gr --tags=nfdhcpd,vpn --name=vpn --backend-id=1 |
437 |
|
438 |
# snf-manage network-list |
439 |
id name flavor owner mac_prefix dhcp state link vms public IPv4 Subnet IPv4 Gateway |
440 |
5 vpn CUSTOM user@grnet.gr bb:00:44 True ACTIVE br200 False 192.168.1.0/24 192.168.1.1 |
441 |
|
442 |
This will add a network in Synnefo DB and create a network in Ganeti backend by |
443 |
issuing: |
444 |
|
445 |
.. code-block:: console |
446 |
|
447 |
# gnt-network add --network=192.168.1.0/24 --gateway=192.168.1.1 --network-type=private --tags=nfdhcpd snf-net-5 |
448 |
|
449 |
# gnt-network connect snf-net-5 default bridged br200 |
450 |
# gnt-network list snf-net-5 |
451 |
Network Subnet Gateway NetworkType MacPrefix GroupList Tags |
452 |
snf-net-5 192.168.1.0/24 192.168.1.1 private bb:00:55 default(bridged, br200) nfdhcpd,private-filtered |
453 |
|
454 |
|