Revision 1a7c1456
b/doc/design-os.rst | ||
---|---|---|
12 | 12 |
============================== |
13 | 13 |
|
14 | 14 |
As of Ganeti 2.10, each instance is associated with an OS definition. An OS |
15 |
definition is a set of scripts (``create``, ``export``, ``import``, ``rename``) |
|
16 |
that are executed with root privileges on the primary host of the instance to |
|
17 |
perform all the OS-related functionality (setting up an operating system inside |
|
18 |
the disks of the instance being created, exporting/importing the instance, |
|
19 |
renaming it). |
|
20 |
|
|
21 |
These scripts receive through environment variables a fixed set of parameters |
|
22 |
related to the instance (such as the hypervisor, the name of the instance, the |
|
23 |
number of disks, and their location) and a set of user defined parameters. |
|
24 |
These parameters are also written in the configuration file of Ganeti, to allow |
|
25 |
future reinstalls of the instance, and in various log files, namely: |
|
15 |
definition is a set of scripts (i.e., ``create``, ``export``, ``import``, |
|
16 |
``rename``) that are executed with root privileges on the primary host of the |
|
17 |
instance. These scripts are responsible for performing all the OS-related |
|
18 |
tasks, namely, create an instance, setup an operating system on the instance's |
|
19 |
disks, export/import the instance, and rename the instance. |
|
20 |
|
|
21 |
These scripts receive, through environment variables, a fixed set of instance |
|
22 |
parameters (such as, the hypervisor, the name of the instance, the number of |
|
23 |
disks and their location) and a set of user defined parameters. Both the |
|
24 |
instance and user defined parameters are written in the configuration file of |
|
25 |
Ganeti, to allow future reinstalls of the instance, and in various log files, |
|
26 |
namely: |
|
26 | 27 |
|
27 | 28 |
* node daemon log file: contains DEBUG strings of the ``/os_validate``, |
28 | 29 |
``/instance_os_add`` and ``/instance_start`` RPC calls. |
... | ... | |
41 | 42 |
|
42 | 43 |
The current situation presents a number of shortcomings: |
43 | 44 |
|
44 |
* Having the installation scripts run as root on the nodes doesn't allow
|
|
45 |
user-defined OS scripts, as they would pose a huge security issue.
|
|
45 |
* Having the installation scripts run as root on the nodes does not allow
|
|
46 |
user-defined OS scripts, as they would pose a huge security risk.
|
|
46 | 47 |
Furthermore, even a script without malicious intentions might end up |
47 |
distrupting a node because of a bug in it.
|
|
48 |
disrupting a node because of due to a bug.
|
|
48 | 49 |
|
49 | 50 |
* Ganeti cannot be used to create instances starting from user provided disk |
50 |
images: even in the (hypothetical) case where the scripts are completely
|
|
51 |
images: even in the (hypothetical) case in which the scripts are completely
|
|
51 | 52 |
secure and run not by root but by an unprivileged user with only the power to |
52 |
mount arbitrary files as disk images, this is a security issue. It has been
|
|
53 |
proven that a carefully crafted file system might exploit kernel |
|
53 |
mount arbitrary files as disk images, this is still a security issue. It has
|
|
54 |
been proven that a carefully crafted file system might exploit kernel
|
|
54 | 55 |
vulnerabilities to gain control of the system. Therefore, directly mounting |
55 | 56 |
images on the Ganeti nodes is not an option. |
56 | 57 |
|
57 | 58 |
* There is no way to inject files into an existing disk image. A common use case |
58 | 59 |
is for the system administrator to provide a standard image of the system, to |
59 | 60 |
be later personalized with the network configuration, private keys identifying |
60 |
the machine, ssh keys of the users and so on. A possible workaround would be |
|
61 |
the machine, ssh keys of the users, and so on. A possible workaround would be
|
|
61 | 62 |
for the scripts to mount the image (only if this is trusted!) and to receive |
62 | 63 |
the configurations and ssh keys as user defined OS parameters. Unfortunately, |
63 | 64 |
this is also not an option for security sensitive material (such as the ssh |
64 | 65 |
keys) because the OS parameters are stored in many places on the system, as |
65 | 66 |
already described above. |
66 | 67 |
|
67 |
* Most other virtualization software simply work with instance images, not with
|
|
68 |
installation scripts. This difference makes the interaction of Ganeti with
|
|
68 |
* Most other virtualization software allow only instance images, but no
|
|
69 |
installation scripts. This difference makes the interaction between Ganeti and
|
|
69 | 70 |
other software difficult. |
70 | 71 |
|
71 | 72 |
Proposed changes |
... | ... | |
74 | 75 |
In order to fix the shortcomings of the current state, we plan to introduce the |
75 | 76 |
following changes. |
76 | 77 |
|
77 |
|
|
78 |
OS parameters categories |
|
79 |
++++++++++++++++++++++++ |
|
78 |
OS parameter categories |
|
79 |
+++++++++++++++++++++++ |
|
80 | 80 |
|
81 | 81 |
Change the OS parameters to have three categories: |
82 | 82 |
|
... | ... | |
99 | 99 |
is an expected and accepted side effect of jobs with secret parameters: if |
100 | 100 |
they fail, they'll have to be restarted manually. |
101 | 101 |
|
102 |
|
|
103 | 102 |
Metadata |
104 | 103 |
++++++++ |
105 | 104 |
|
... | ... | |
110 | 109 |
access exclusively to its own metadata, and it will be only able to communicate |
111 | 110 |
with its host over this channel. This is the approach followed the |
112 | 111 |
``cloud-init`` tool and more details will be provided in the `Communication |
113 |
mechanism and metadata service`_ section. |
|
114 |
|
|
112 |
mechanism`_ and `Metadata service`_ sections. |
|
115 | 113 |
|
116 | 114 |
Installation procedure |
117 | 115 |
++++++++++++++++++++++ |
118 | 116 |
|
119 |
A new installation procedure will be introduced, with which it will be possible |
|
120 |
to use an installation medium and run the OS scripts in an optional virtualized |
|
121 |
environment and with an optional personalization package. There will be two |
|
122 |
sets of parameters, namely, installation parameters, which are used mainly for |
|
123 |
installs and reinstalls, and execution parameters, which are used in all the |
|
124 |
other runs that are not part of an installation procedure. |
|
125 |
|
|
126 |
This set of installation parameters will allow, e.g., to attach an installation |
|
127 |
floppy/cdrom/network, change the boot device order, or specify a disk image to |
|
128 |
be used. Through this set of parameters, the administrator will have to provide |
|
129 |
the hypervisor a location for an installation medium for the instance (e.g., a |
|
130 |
boot disk, a network image, etc). This medium will carry out the installation |
|
131 |
of the instance onto the instance's disks and will then be responsible for |
|
132 |
getting the parameters for configuring the instance, such as, network |
|
133 |
interfaces, IP address, and hostname. These parameters are taken from the |
|
134 |
metadata. The installation parameters will be stored in the configuration of |
|
135 |
Ganeti and used in future reinstalls, but not during normal execution. |
|
117 |
A new installation procedure will be introduced. There will be two sets of |
|
118 |
parameters, namely, installation parameters, which are used mainly for installs |
|
119 |
and reinstalls, and execution parameters, which are used in all the other runs |
|
120 |
that are not part of an installation procedure. Also, it will be possible to |
|
121 |
use an installation medium and/or run the OS scripts in an optional virtualized |
|
122 |
environment, and optionally use a personalization package. This section details |
|
123 |
all of these options. |
|
124 |
|
|
125 |
The set of installation parameters will allow, for example, to attach an |
|
126 |
installation floppy/cdrom/network, change the boot device order, or specify a |
|
127 |
disk image to be used. Through this set of parameters, the administrator will |
|
128 |
have to provide the hypervisor a location for an installation medium for the |
|
129 |
instance (e.g., a boot disk, a network image, etc). This medium will carry out |
|
130 |
the installation of the instance onto the instance's disks and will then be |
|
131 |
responsible for getting the parameters for configuring the instance, such as, |
|
132 |
network interfaces, IP address, and hostname. These parameters are taken from |
|
133 |
the metadata. The installation parameters will be stored in the configuration |
|
134 |
of Ganeti and used in future reinstalls, but not during normal execution. |
|
136 | 135 |
|
137 | 136 |
The instance is reinstalled using the same installation parameters from the |
138 | 137 |
first installation. However, it will be the administrator's responsibility to |
139 |
ensure that the any installation media is still available at the proper location
|
|
138 |
ensure that the installation media is still available at the proper location |
|
140 | 139 |
when a reinstall occurs. |
141 | 140 |
|
142 | 141 |
The parameter ``--os-parameters`` can still be used to specify the OS |
143 | 142 |
parameters. However, without OS scripts, Ganeti cannot do more than a syntactic |
144 |
check to validate the supplied OS parameters string. As a result, this string |
|
145 |
will be directly passed to the instance as part of the metadata. If the |
|
146 |
installation procedure is running inside a virtualized environment, then Ganeti |
|
147 |
will take these parameters from the metadata and pass them to the OS scripts as |
|
148 |
environment variables. |
|
143 |
check to validate the supplied OS parameter string. As a result, this string |
|
144 |
will be passed directly to the instance as part of the metadata. If OS scripts |
|
145 |
are used and the installation procedure is running inside a virtualized |
|
146 |
environment, Ganeti will take these parameters from the metadata and pass them |
|
147 |
to the OS scripts as environment variables. |
|
148 |
|
|
149 |
Ganeti allows the following installation options: |
|
149 | 150 |
|
150 | 151 |
* Use a disk image: |
151 | 152 |
|
... | ... | |
159 | 160 |
|
160 | 161 |
The parameter ``--os-type`` (short version: ``-o``), is currently used to |
161 | 162 |
specify the OS scripts. This parameter will still be used to specify the OS |
162 |
scripts with the difference that these OS scripts may optionally run inside a
|
|
163 |
scripts with the difference that these scripts may optionally run inside a |
|
163 | 164 |
virtualized environment for safety reasons, depending on whether they are |
164 | 165 |
trusted or not. For more details on trusted and untrusted OS scripts, refer |
165 |
to the `Installation process in a virtualized environment`_ section. |
|
166 |
to the `Installation process in a virtualized environment`_ section. Note |
|
167 |
that this parameter will become optional thus allowing a user to create an |
|
168 |
instance specifying only, for example, a disk image or a cdrom image to boot |
|
169 |
from. |
|
166 | 170 |
|
167 | 171 |
* Personalization package |
168 | 172 |
|
... | ... | |
170 | 174 |
URL for a "personalization package", which is an archive containing a set of |
171 | 175 |
files meant to be overlayed on top of the OS file system at the end of the |
172 | 176 |
setup process and before the VM is started for the first time in normal mode. |
173 |
Ganeti will provide a mechanism for receiving and unpacking this archive |
|
174 |
whether the installation is being performed inside the virtualized environment
|
|
175 |
or not. |
|
177 |
Ganeti will provide a mechanism for receiving and unpacking this archive,
|
|
178 |
independently of whether the installation is being performed inside the
|
|
179 |
virtualized environment or not.
|
|
176 | 180 |
|
177 | 181 |
The archive will be in TAR-GZIP format (with extension ``.tar.gz`` or |
178 | 182 |
``.tgz``) and contain the files according to the directory structure that will |
... | ... | |
207 | 211 |
* Combine a disk image, OS scripts, and a personalization package |
208 | 212 |
|
209 | 213 |
It will possible to combine a disk image, OS scripts, and a personalization |
210 |
package, both with or without a virtualized environment. There is one |
|
211 |
exception which is if there are untrusted OS scripts. At least, an |
|
212 |
installation medium or OS scripts should be specified. |
|
214 |
package, both with or without a virtualized environment (see the exception |
|
215 |
below). At least, an installation medium or OS scripts should be specified. |
|
213 | 216 |
|
214 | 217 |
The disk image of the actual virtual appliance, which bootstraps the virtual |
215 | 218 |
environment used in the installation procedure, will be read only, so that a |
... | ... | |
242 | 245 |
written in the `Proposed changes`_ Section. Extra details will be provided in |
243 | 246 |
the following subsections. |
244 | 247 |
|
245 |
Communication mechanism and metadata service |
|
246 |
++++++++++++++++++++++++++++++++++++++++++++ |
|
247 |
|
|
248 |
The communication mechanism and the metadata service are described together |
|
249 |
because they are deeply tied. The communication mechanism will be made more |
|
250 |
generic because it can be used for other purposes in the future (like allowing |
|
251 |
instances to explicitly send commands to Ganeti, or to let Ganeti control a |
|
252 |
helper instance, like the one hereby introduced for performing OS installs |
|
253 |
inside a safe environment). |
|
248 |
Communication mechanism |
|
249 |
+++++++++++++++++++++++ |
|
250 |
|
|
251 |
The communication mechanism will be an exclusive, generic, bidirectional |
|
252 |
communication channel between Ganeti hosts and guests. |
|
253 |
|
|
254 |
exclusive |
|
255 |
The communication mechanism allows communication between a guest and its host, |
|
256 |
but it does not allow a guest to communicate with other guests or reach the |
|
257 |
outside world. |
|
258 |
|
|
259 |
generic |
|
260 |
The communication mechanism allows a guest to reach any service on the host, |
|
261 |
not just the metadata service. Examples of valid communication include, but |
|
262 |
are not limited to, access to the metadata service, send commands to Ganeti, |
|
263 |
request changes to parameters, such as, those related to the distribution |
|
264 |
upgrades, and let Ganeti control a helper instance, such as, the one for |
|
265 |
performing OS installs inside a safe environment. |
|
266 |
|
|
267 |
bidirectional |
|
268 |
The communication mechanism allows communication to be initiated from either |
|
269 |
party, namely, from a host to a guest or guest to host. |
|
270 |
|
|
271 |
Note that Ganeti will allow communication with any service (e.g., daemon) running |
|
272 |
on the host and, as a result, Ganeti will not be responsible for ensuring that |
|
273 |
only the metadata service is reachable. It is the responsibility of each system |
|
274 |
administrator to ensure that the extra firewalling and routing rules specified |
|
275 |
on the host provide the necessary protection on a given Ganeti installation and, |
|
276 |
at the same time, do not accidentally override the behaviour hereby described |
|
277 |
which makes the communication between the host and the guest exclusive, generic, |
|
278 |
and bidirectional, unless intended. |
|
254 | 279 |
|
255 | 280 |
The communication mechanism will be enabled automatically during an installation |
256 |
procedure that requires a virtualized environment, but for backwards |
|
257 |
compatibility it will be disabled when the instance is running normally, unless |
|
258 |
it is explicitly requested. Specifically, a new parameter |
|
259 |
``--communication=yes|no`` (short version: ``-C``) will be added to |
|
260 |
``gnt-instance add`` and ``gnt-instance modify``. It will determine whether the |
|
261 |
instance has a communication channel set to interact with the host and receive |
|
262 |
metadata. The value of this parameter will be saved as part of the configuration |
|
263 |
of the instance. |
|
264 |
|
|
265 |
When the communication mechanism is enabled, Ganeti will create a new network |
|
266 |
interface inside the instance. This additional network interface will be the |
|
267 |
last one in the instance, after all the user defined ones. On the host side, |
|
268 |
this interface will only be accessible to the host itself, and not routed |
|
269 |
outside the machine. |
|
270 |
On this network interface, the instance will connect using the IP: |
|
271 |
169.254.169.253 and netmask 255.255.255.0. |
|
272 |
The host will be on the same network, with the IP address: 169.254.169.254. |
|
273 |
|
|
274 |
The way to create this interface depends on the specific hypervisor being used. |
|
275 |
In KVM, it is possible to create a network interface inside the instance without |
|
276 |
having a corresponding interface created on the host. Using a command like:: |
|
277 |
|
|
278 |
kvm -net nic -net \ |
|
279 |
user,restrict=on,net=169.254.169.0/24,host=169.254.169.253, |
|
280 |
guestfwd=tcp:169.254.169.254:80-tcp:127.0.0.1:8080 |
|
281 |
|
|
282 |
a network interface will be created inside the VM, part of the 169.254.169.0/24 |
|
283 |
network, where the VM will have IP address .253 and the host port 8080 will be |
|
284 |
reachable on port 80. |
|
285 |
|
|
286 |
In Xen, unfortunately, such a capability is not present, and an actual network |
|
287 |
interface has to be created on the host (using the ``vif`` parameter of the Xen |
|
288 |
configuration file). Each instance will have its corresponding ``vif`` network |
|
289 |
interface on the host. These interfaces will not be connected to each other in |
|
290 |
any way, and Ganeti will not configure them to allow traffic to be forwarded |
|
291 |
beyond the host machine. The ``vif-route`` script of Xen might be helpful in |
|
292 |
implementing this. |
|
293 |
It will be the system administrator's responsibility to ensure that the extra |
|
294 |
firewalling and routing rules specified on the host don't allow this |
|
295 |
accidentally. |
|
296 |
|
|
297 |
The instance will be able to connect to 169.254.169.254:80, and issue GET |
|
298 |
requests to an HTTP server that will provide the instance metadata. |
|
299 |
|
|
300 |
The choice of this IP address and port for accessing the metadata is done for |
|
301 |
compatibility reasons with OpenStack's and Amazon EC2's ways of providing |
|
302 |
metadata to the instance. The metadata will be provided by a single daemon, |
|
303 |
which will determine what instance the request comes from and reply with the |
|
304 |
metadata specific for that instance. |
|
281 |
procedure that requires a virtualized environment, but, for backwards |
|
282 |
compatibility, it will be disabled when the instance is running normally, unless |
|
283 |
explicitly requested. Specifically, a new parameter ``--communication=yes|no`` |
|
284 |
(short version: ``-C``) will be added to ``gnt-instance add`` and ``gnt-instance |
|
285 |
modify``. This parameter will determine whether the communication mechanism is |
|
286 |
enabled for a particular instance. The value of this parameter will be saved as |
|
287 |
part of the instance's configuration. |
|
288 |
|
|
289 |
The communication mechanism will be implemented through network interfaces on |
|
290 |
the host and the guest, and Ganeti will be responsible for the host side, |
|
291 |
namely, creating a TAP interface for each guest and configuring these interfaces |
|
292 |
to have IP address ``169.254.169.254`` and netmask ``255.255.255.255``. This |
|
293 |
network interface will be connected to the guest's last network interface, which |
|
294 |
is meant to be used exclusively for the communication mechanism and is defined |
|
295 |
after all the used-defined interfaces. The last interface was chosen (as |
|
296 |
opposed to the first one, for example) because the first interface is generally |
|
297 |
understood and the main gateway out, and also because it minimizes the impact on |
|
298 |
existing systems, for example, in a scenario where the system administrator has |
|
299 |
a running cluster and wants to enable the communication mechanism for already |
|
300 |
existing instances, which might have been created with older versions of Ganeti. |
|
301 |
Further, DBus should assist in keeping the guest network interfaces more stable. |
|
302 |
|
|
303 |
On the guest side, each instance will have its own MAC address and IP address. |
|
304 |
Both the guest's MAC address and IP address must be unique within a single |
|
305 |
cluster. An IP is unique within a single cluster, and not within a single host, |
|
306 |
in order to minimize disruption of connectivity, for example, during live |
|
307 |
migration, in particular since an instance is not aware when it changes host. |
|
308 |
Unfortunately, a side-effect of this decision is that a cluster can have a |
|
309 |
maximum of a ``/16`` network allowed instances (with communication enabled). If |
|
310 |
necessary to overcome this limit, it should be possible to allow different |
|
311 |
networks to be configured link-local only. |
|
312 |
|
|
313 |
The guest will use the DHCP protocol on its last network interface to contact a |
|
314 |
DHCP server running on the host and thus determine its IP address. The DHCP |
|
315 |
server is configured, started, and stopped, by Ganeti and it will be listening |
|
316 |
exclusively on the TAP network interfaces of the guests in order not to |
|
317 |
interfere with a potential DHCP server running on the same host. Furthermore, |
|
318 |
the DHCP server will only recognize MAC and IP address pairs that have been |
|
319 |
approved by Ganeti. |
|
320 |
|
|
321 |
The TAP network interfaces created for each guest share the same IP address. |
|
322 |
Therefore, it will be necessary to extend the routing table with rules specific |
|
323 |
to each guest. This can be achieved with the following command, which takes the |
|
324 |
guest's unique IP address and its TAP interface:: |
|
325 |
|
|
326 |
route add -host <ip> dev <ifname> |
|
327 |
|
|
328 |
This rule has the additional advantage of preventing guests from trying to lease |
|
329 |
IP addresses from the DHCP server other than the own that has been assigned to |
|
330 |
them by Ganeti. The guest could lie about its MAC address to the DHCP server |
|
331 |
and try to steal another guest's IP address, however, this routing rule will |
|
332 |
block traffic (i.e., IP packets carrying the wrong IP) from the DHCP server to |
|
333 |
the malicious guest. Similarly, the guest could lie about its IP address (i.e., |
|
334 |
simply assign a predefined IP address, perhaps from another guest), however, |
|
335 |
replies from the host will not be routed to the malicious guest. |
|
336 |
|
|
337 |
This routing rule ensures that the communication channel is exclusive but, as |
|
338 |
mentioned before, it will not prevent guests from accessing any service on the |
|
339 |
host. It is the system administrator's responsibility to employ the necessary |
|
340 |
``iptables`` rules. In order to achieve this, Ganeti will provide ``ifup`` |
|
341 |
hooks associated with the guest network interfaces which will give system |
|
342 |
administrator's the opportunity to customize their own ``iptables``, if |
|
343 |
necessary. Ganeti will also provide examples of such hooks. However, these are |
|
344 |
meant to personalized to each Ganeti installation and not to be taken as |
|
345 |
production ready scripts. |
|
346 |
|
|
347 |
For KVM, an instance will be started with a unique MAC address and the file |
|
348 |
descriptor for the TAP network interface meant to be used by the communication |
|
349 |
mechanism. Ganeti will be responsible for generating a unique MAC address for |
|
350 |
the guest, opening the TAP interface, and passing its file descriptor to KVM:: |
|
351 |
|
|
352 |
kvm -net nic,macaddr=<mac> -net tap,fd=<tap-fd> ... |
|
353 |
|
|
354 |
For Xen, a network interface will be created on the host (using the ``vif`` |
|
355 |
parameter of the Xen configuration file). Each instance will have its |
|
356 |
corresponding ``vif`` network interface on the host. The ``vif-route`` script |
|
357 |
of Xen might be helpful in implementing this. |
|
358 |
|
|
359 |
Metadata service |
|
360 |
++++++++++++++++ |
|
361 |
|
|
362 |
An instance will be able to reach metadata service on ``169.254.169.254:80`` in |
|
363 |
order to, for example, retrieve its metadata. This IP address and port were |
|
364 |
chosen for compatibility with the OpenStack and Amazon EC2 metadata service. |
|
365 |
The metadata service will be provided by a single daemon, which will determine |
|
366 |
the source instance for a given request and reply with the metadata pertaining |
|
367 |
to that instance. |
|
305 | 368 |
|
306 | 369 |
Where possible, the metadata will be provided in a way compatible with Amazon |
307 | 370 |
EC2, at:: |
308 | 371 |
|
309 | 372 |
http://169.254.169.254/<version>/meta-data/* |
310 | 373 |
|
311 |
If some metadata are Ganeti-specific and don't fit this structure, they will be
|
|
312 |
provided at::
|
|
374 |
Ganeti-specific metadata, that does not fit this structure, will be provided
|
|
375 |
at:: |
|
313 | 376 |
|
314 | 377 |
http://169.254.169.254/ganeti/<version>/meta_data.json |
315 | 378 |
|
316 |
``<version>`` is either a date in YYYY-MM-DD format, or ``latest`` to indicate
|
|
317 |
the most recent available protocol version. |
|
379 |
where ``<version>`` is either a date in YYYY-MM-DD format, or ``latest`` to
|
|
380 |
indicate the most recent available protocol version.
|
|
318 | 381 |
|
319 | 382 |
If needed in the future, this structure also allows us to support OpenStack's |
320 | 383 |
metadata at:: |
321 | 384 |
|
322 | 385 |
http://169.254.169.254/openstack/<version>/meta_data.json |
323 | 386 |
|
324 |
A bi-directional, pipe-like communication channel will be provided. The instance
|
|
325 |
will be able to receive data from the host by a GET request at:: |
|
387 |
A bi-directional, pipe-like communication channel will also be provided. The
|
|
388 |
instance will be able to receive data from the host by a GET request at::
|
|
326 | 389 |
|
327 | 390 |
http://169.254.169.254/ganeti/<version>/read |
328 | 391 |
|
... | ... | |
331 | 394 |
http://169.254.169.254/ganeti/<version>/write |
332 | 395 |
|
333 | 396 |
As in a pipe, once the data are read, they will not be in the buffer anymore, so |
334 |
subsequent GET requests to ``read`` will not return the same data twice. |
|
335 |
Unlike a pipe, though, it will not be possible to perform blocking I/O |
|
336 |
operations. |
|
397 |
subsequent GET requests to ``read`` will not return the same data. However, |
|
398 |
unlike a pipe, it will not be possible to perform blocking I/O operations. |
|
337 | 399 |
|
338 |
The OS parameters will be accessible through a GET |
|
339 |
request at:: |
|
400 |
The OS parameters will be accessible through a GET request at:: |
|
340 | 401 |
|
341 | 402 |
http://169.254.169.254/ganeti/<version>/os/parameters.json |
342 | 403 |
|
... | ... | |
352 | 413 |
|
353 | 414 |
where ``<script_name>`` is the name of the script. |
354 | 415 |
|
355 |
|
|
356 | 416 |
Rationale |
357 | 417 |
--------- |
358 | 418 |
|
... | ... | |
366 | 426 |
in the user interface, whereas it is fairly common to have an unconfigured |
367 | 427 |
network interface in a system, usually without any negative side effects. |
368 | 428 |
|
369 |
|
|
370 | 429 |
Installation process in a virtualized environment |
371 | 430 |
+++++++++++++++++++++++++++++++++++++++++++++++++ |
372 | 431 |
|
... | ... | |
407 | 466 |
Ganeti will provide a script to be run at install time that can be used to |
408 | 467 |
create the virtualized environment that will perform the OS installation of new |
409 | 468 |
instances. |
410 |
This script will build a debootstrapped basic debian system including a software
|
|
469 |
This script will build a debootstrapped basic Debian system including a software
|
|
411 | 470 |
that will read the metadata, setup the environment variables and launch the |
412 | 471 |
installation scripts inside the virtualized environment. The script will also |
413 | 472 |
provide hooks for personalization. |
... | ... | |
416 | 475 |
long as they connect to Ganeti over the described communication mechanism and |
417 | 476 |
they know how to read and use the provided metadata to create a new instance. |
418 | 477 |
|
419 |
While performing an installation in the virtualized environment, a |
|
420 |
personalizable timeout will be used to detect possible problems with the |
|
421 |
installation process, and to kill the virtualized environment. The timeout will |
|
422 |
be optional and set on a cluster basis by the administrator. If set, it will be |
|
423 |
the total time allowed to setup an instance inside the appliance. It is mainly |
|
424 |
meant as a safety measure to prevent an instance taken over by malicious scripts |
|
425 |
to be available for a long time. |
|
426 |
|
|
427 |
.. vim: set textwidth=72 : |
|
428 |
.. Local Variables: |
|
429 |
.. mode: rst |
|
430 |
.. fill-column: 72 |
|
431 |
.. End: |
|
478 |
While performing an installation in the virtualized environment, a customizable |
|
479 |
timeout will be used to detect possible problems with the installation process, |
|
480 |
and to kill the virtualized environment. The timeout will be optional and set on |
|
481 |
a cluster basis by the administrator. If set, it will be the total time allowed |
|
482 |
to setup an instance inside the appliance. It is mainly meant as a safety |
|
483 |
measure to prevent an instance taken over by malicious scripts to be available |
|
484 |
for a long time. |
|
485 |
|
|
486 |
Alternatives to design and implementation |
|
487 |
========================================= |
|
488 |
|
|
489 |
This section lists alternatives to design and implementation, which came up |
|
490 |
during the development of this design document, that will not be implemented. |
|
491 |
Please read carefully through the limitations and security concerns of each of |
|
492 |
these alternatives. |
|
493 |
|
|
494 |
Port forwarding in KVM |
|
495 |
++++++++++++++++++++++ |
|
496 |
|
|
497 |
The communication mechanism could have been implemented in KVM using guest port |
|
498 |
forwarding, as opposed to network interfaces. There are two alternatives in |
|
499 |
KVM's guest port forwarding, namely, creating a forwarding device, such as, a |
|
500 |
TCP/IP connection, or executing a command. However, we have determined that |
|
501 |
both of these options are not viable. |
|
502 |
|
|
503 |
A TCP/IP forwarding device can be created through the following KVM invocation:: |
|
504 |
|
|
505 |
kvm -net nic -net \ |
|
506 |
user,restrict=on,net=169.254.0.0/16,host=169.254.169.253, |
|
507 |
guestfwd=tcp:169.254.169.254:80-tcp:127.0.0.1:8080 ... |
|
508 |
|
|
509 |
This invocation even has the advantage that it can block undesired traffic |
|
510 |
(i.e., traffic that is not explicitly specified in the arguments) and it can |
|
511 |
remap ports, which would have allowed the metadata service daemon to run in port |
|
512 |
8080 instead of 80. However, in this scheme, KVM opens the TCP connection only |
|
513 |
once, when it is started, and, if the connection breaks, KVM will not |
|
514 |
reestablish the connection. Furthermore, opening the TCP connection only once |
|
515 |
interferes with the HTTP protocol, which needs to dynamically establish and |
|
516 |
close connections. |
|
517 |
|
|
518 |
The alternative to the TCP/IP forwarding device is to execute a command. The |
|
519 |
KVM invocation for this is, for example, the following:: |
|
520 |
|
|
521 |
kvm -net nic -net \ |
|
522 |
"user,restrict=on,net=169.254.0.0/16,host=169.254.169.253, |
|
523 |
guestfwd=tcp:169.254.169.254:80-netcat 127.0.0.1 8080" ... |
|
524 |
|
|
525 |
The advantage of this approach is that the command is executed each time the |
|
526 |
guest initiates a connection. This is the ideal situation, however, it is only |
|
527 |
supported in KVM 1.2 and above, and, therefore, not viable because we want to |
|
528 |
provide support for at least KVM version 1.0, which is the version provided by |
|
529 |
Ubuntu LTS. |
|
530 |
|
|
531 |
Alternatives to the DHCP server |
|
532 |
+++++++++++++++++++++++++++++++ |
|
533 |
|
|
534 |
There are alternatives to using the DHCP server, for example, by assigning a |
|
535 |
fixed IP address to guests, such as, the IP address ``169.254.169.253``. |
|
536 |
However, this introduces a routing problem, namely, how to route incoming |
|
537 |
packets from the same source IP to the host. This problem can be overcome in a |
|
538 |
number of ways. |
|
539 |
|
|
540 |
The first solution is to use NAT to translate the incoming guest IP address, for |
|
541 |
example, ``169.254.169.253``, to a unique IP address, for example, |
|
542 |
``169.254.0.1``. Given that NAT through ``ip rule`` is deprecated, users can |
|
543 |
resort to ``iptables``. Note that this has not yet been tested. |
|
544 |
|
|
545 |
Another option, which has been tested, but only in a prototype, is to connect |
|
546 |
the TAP network interfaces of the guests to a bridge. The bridge takes the |
|
547 |
configuration from the TAP network interfaces, namely, IP address |
|
548 |
``169.254.169.254`` and netmask ``255.255.255.255``, thus leaving those |
|
549 |
interfaces without an IP address. Note that in this setting, guests will be |
|
550 |
able to reach each other, therefore, if necessary, additional ``iptables`` rules |
|
551 |
can be put in place to prevent it. |
Also available in: Unified diff