Statistics
| Branch: | Tag: | Revision:

root / doc / source / install.rst @ 6de88ee1

History | View | Annotate | Download (19.4 kB)

1
************
2
Installation
3
************
4

    
5
.. toctree::
6
    :maxdepth: 2
7

    
8
Debian Wheezy (x64) - Django 1.4.x
9
==================================
10
This guide assumes that installation is carried out in /srv/flowspy directory. If other directory is to be used, please change the corresponding configuration files. It is also assumed that the root user will perform every action.
11

    
12

    
13
Upgrading from v<1.1.x
14
----------------------
15

    
16
.. note::
17
    If PEER\_\*\_TABLE tables are set to FALSE in settings.py, you need to perform the south migrations per application::
18

    
19
        ./manage.py migrate longerusername
20
        ./manage.py migrate flowspec
21
        ./manage.py migrate accounts
22

    
23

    
24

    
25
If upgrading from flowspy version <1.1.x pay attention to settings.py changes. Also, do not forget to run if PEER\_\*\_TABLE tables are set to TRUE in settings.py::
26

    
27
    ./manage.py migrate
28

    
29
to catch-up with latest database changes.
30

    
31
Upgrading from v<1.0.x
32
----------------------
33
If upgrading from flowspy version <1.0.x pay attention to settings.py changes. Also, do not forget to run::
34

    
35
    ./manage.py migrate
36

    
37
to catch-up with latest database changes.
38

    
39
Required system packages
40
------------------------
41
Update and install the required packages::
42

    
43
    apt-get update
44
    apt-get upgrade
45
    apt-get install mysql-server apache2 memcached libapache2-mod-proxy-html gunicorn beanstalkd python-django python-django-south python-django-tinymce tinymce python-mysqldb python-yaml python-memcache python-django-registration python-ipaddr python-lxml mysql-client git python-django-celery python-paramiko python-gevent vim
46

    
47
Also, django rest framework package is required. In debian Wheezy it is not available, but one can install it via pip.
48

    
49
.. note::
50
    Set username and password for mysql if used
51

    
52
.. note::
53
    If you wish to deploy an outgoing mail server, now it is time to do it. Otherwise you could set FoD to send out mails via a third party account
54

    
55
Create a database
56
-----------------
57
If you are using mysql, you should create a database::
58

    
59
    mysql -u root -p -e 'create database fod'
60

    
61
Required application packages
62
-----------------------------
63
Get the required packages and their dependencies and install them::
64

    
65
    apt-get install libxml2-dev libxslt-dev gcc python-dev
66

    
67
- ncclient: NETCONF python client::
68

    
69
    cd ~
70
    git clone https://github.com/leopoul/ncclient.git
71
    cd ncclient
72
    python setup.py install
73

    
74
- nxpy: Python Objects from/to XML proxy::
75

    
76
    cd ~
77
    git clone https://code.grnet.gr/git/nxpy
78
    cd nxpy
79
    python setup.py install
80

    
81
- flowspy: core application. Installation is done at /srv/flowspy::
82

    
83
    cd /srv
84
    git clone https://code.grnet.gr/git/flowspy
85
    cd flowspy
86

    
87
Application configuration
88
=========================
89
Copy settings.py.dist to settings.py::
90

    
91
    cd flowspy
92
    cp settings.py.dist settings.py
93

    
94
Edit settings.py file and set the following according to your configuration::
95

    
96
    ADMINS: set your admin name and email (assuming that your server can send notifications)
97
    DATABASES (to point to your local database). You could use views instead of tables for models: peer, peercontacts, peernetworks. For this to work we suggest MySQL with MyISAM db engine
98
    SECRET_KEY : Make this unique, and don't share it with anybody
99
    STATIC_ROOT: /srv/flowspy/static (or your installation directory)
100
    STATIC_URL (static media directory) . If you have followed the above this should be: /srv/flowspy/static
101
    TEMPLATE_DIRS : If you have followed the above this should be: /srv/flowspy/templates
102
    CACHE_BACKEND:  Enable Memcached for production or leave to DummyCache for development environments
103
    Alternatively you could go for redis with the corresponding Django client lib.
104
    NETCONF_DEVICE (tested with Juniper EX4200 but any BGP enabled Juniper should work). This is the flowspec capable device
105
    NETCONF_USER (enable ssh and netconf on device)
106
    NETCONF_PASS
107
    If beanstalk is selected the following should be left intact.
108
    BROKER_HOST (beanstalk host)
109
    BROKER_PORT (beanstalk port)
110
    SERVER_EMAIL
111
    EMAIL_SUBJECT_PREFIX
112
    If beanstalk is selected the following should be left intact.
113
    BROKER_URL (beanstalk url)
114
    SHIB_AUTH_ENTITLEMENT (if you go for Shibboleth authentication)
115
    NOTIFY_ADMIN_MAILS (bcc mail addresses)
116
    PROTECTED_SUBNETS (subnets for which source or destination address will prevent rule creation and notify the NOTIFY_ADMIN_MAILS)
117
    The whois client is meant to be used in case you have inserted peers with their ASes in the peers table and wish to get network info for each one in an automated manner.
118
    PRIMARY_WHOIS
119
    ALTERNATE_WHOIS
120
    If you wish to deploy FoD with Shibboleth change the following attributes according to your setup:
121
    SHIB_AUTH_ENTITLEMENT = 'urn:mace'
122
    SHIB_ADMIN_DOMAIN = 'example.com'
123
    SHIB_LOGOUT_URL = 'https://example.com/Shibboleth.sso/Logout'
124
    SHIB_USERNAME = ['HTTP_EPPN']
125
    SHIB_MAIL = ['mail', 'HTTP_MAIL', 'HTTP_SHIB_INETORGPERSON_MAIL']
126
    SHIB_FIRSTNAME = ['HTTP_SHIB_INETORGPERSON_GIVENNAME']
127
    SHIB_LASTNAME = ['HTTP_SHIB_PERSON_SURNAME']
128
    SHIB_ENTITLEMENT = ['HTTP_SHIB_EP_ENTITLEMENT']
129

    
130
If you have not installed an outgoing mail server you can always use your own account (either corporate or gmail, hotmail ,etc) by adding the following lines in settings.py::
131

    
132
    EMAIL_USE_TLS = True #(or False)
133
    EMAIL_HOST = 'smtp.example.com'
134
    EMAIL_HOST_USER = 'username'
135
    EMAIL_HOST_PASSWORD = 'yourpassword'
136
    EMAIL_PORT = 587 #(outgoing)
137

    
138
It is strongly advised that you do not change the following to False values unless, you want to integrate FoD with you CRM or members database. This implies that you are able/have the rights to create database views between the two databases::
139

    
140
    PEER_MANAGED_TABLE = True
141
    PEER_RANGE_MANAGED_TABLE = True
142
    PEER_TECHC_MANAGED_TABLE = True
143

    
144
By doing that the corresponding tables as defined in peers/models will not be created. As noted above, you have to create the views that the tables will rely on.
145

    
146
.. note::
147
    Soon we will release a version with django-registration as a means to add users and Shibboleth will become an alternative
148

    
149
Let's move on with some copies and dir creations::
150

    
151
    mkdir /var/log/fod
152
    chown www-data.www-data /var/log/fod
153
    cp urls.py.dist urls.py
154
    cd ..
155

    
156
.. note::
157
    LOG_FILE_LOCATION in settings.py is set to **/var/log/fod**. Adjust the chown command above to your selected dir.
158

    
159
System configuration
160
====================
161
Apache operates as a gunicorn Proxy with WSGI and Shibboleth modules enabled.
162
Depending on the setup the apache configuration may vary::
163

    
164
    a2enmod rewrite
165
    a2enmod proxy
166
    a2enmod ssl
167
    a2enmod proxy_http
168

    
169
If shibboleth is to be used::
170

    
171
    apt-get install libapache2-mod-shib2
172
    a2enmod shib2
173

    
174
Now it is time to configure beanstalk, gunicorn, celery and apache.
175

    
176
beanstalkd
177
----------
178
Enable beanstalk by editting /etc/default/beanstalkd::
179

    
180
    vim /etc/default/beanstalkd
181

    
182
Uncomment the line **START=yes** to enable beanstalk
183

    
184
Start beanstalkd::
185

    
186
    service beanstalkd start
187

    
188
gunicorn.d
189
----------
190
Create and edit /etc/gunicorn.d/fod::
191

    
192
    vim /etc/gunicorn.d/fod
193

    
194
FoD is served via gunicorn and is then proxied by Apache. If the above directory conventions have been followed so far, then your configuration should be::
195

    
196
    CONFIG = {
197
          'mode': 'django',
198
          'working_dir': '/srv/flowspy',
199
          'args': (
200
               '--bind=127.0.0.1:8081',
201
               '--workers=1',
202
               '--worker-class=egg:gunicorn#gevent',
203
               '--timeout=30',
204
               '--debug',
205
               '--log-level=debug',
206
               '--log-file=/var/log/gunicorn/fod.log',
207
          ),
208
    }
209

    
210

    
211
celeryd
212
-------
213
Celery is used over beanstalkd to apply firewall rules in a serial manner so that locks are avoided on the flowspec capable device. In our setup celery runs via django. That is why the python-django-celery package was installed.
214

    
215
Create the celeryd daemon at /etc/init.d/celeryd **if it does not already exist**::
216

    
217
    vim /etc/init.d/celeryd
218

    
219
The configuration should be::
220

    
221
    #!/bin/sh -e
222
    # ============================================
223
    #  celeryd - Starts the Celery worker daemon.
224
    # ============================================
225
    #
226
    # :Usage: /etc/init.d/celeryd {start|stop|force-reload|restart|try-restart|status}
227
    # :Configuration file: /etc/default/celeryd
228
    #
229
    # See http://docs.celeryq.org/en/latest/cookbook/daemonizing.html#init-script-celeryd
230

    
231

    
232
    ### BEGIN INIT INFO
233
    # Provides:              celeryd
234
    # Required-Start:     $network $local_fs $remote_fs
235
    # Required-Stop:       $network $local_fs $remote_fs
236
    # Default-Start:       2 3 4 5
237
    # Default-Stop:        0 1 6
238
    # Short-Description: celery task worker daemon
239
    # Description:          Starts the Celery worker daemon for a single project.
240
    ### END INIT INFO
241

    
242
    #set -e
243

    
244
    DEFAULT_PID_FILE="/var/run/celery/%n.pid"
245
    DEFAULT_LOG_FILE="/var/log/celery/%n.log"
246
    DEFAULT_LOG_LEVEL="INFO"
247
    DEFAULT_NODES="celery"
248
    DEFAULT_CELERYD="-m celery.bin.celeryd_detach"
249
    ENABLED="false"
250

    
251
    [ -r "$CELERY_DEFAULTS" ] && . "$CELERY_DEFAULTS"
252

    
253
    [ -r /etc/default/celeryd ] && . /etc/default/celeryd
254

    
255
    if [ "$ENABLED" != "true" ]; then
256
          echo "celery daemon disabled - see /etc/default/celeryd."
257
          exit 0
258
    fi
259

    
260

    
261
    CELERYD_PID_FILE=${CELERYD_PID_FILE:-${CELERYD_PIDFILE:-$DEFAULT_PID_FILE}}
262
    CELERYD_LOG_FILE=${CELERYD_LOG_FILE:-${CELERYD_LOGFILE:-$DEFAULT_LOG_FILE}}
263
    CELERYD_LOG_LEVEL=${CELERYD_LOG_LEVEL:-${CELERYD_LOGLEVEL:-$DEFAULT_LOG_LEVEL}}
264
    CELERYD_MULTI=${CELERYD_MULTI:-"celeryd-multi"}
265
    CELERYD=${CELERYD:-$DEFAULT_CELERYD}
266
    CELERYCTL=${CELERYCTL:="celeryctl"}
267
    CELERYD_NODES=${CELERYD_NODES:-$DEFAULT_NODES}
268

    
269
    export CELERY_LOADER
270

    
271
    if [ -n "$2" ]; then
272
          CELERYD_OPTS="$CELERYD_OPTS $2"
273
    fi
274

    
275
    CELERYD_LOG_DIR=`dirname $CELERYD_LOG_FILE`
276
    CELERYD_PID_DIR=`dirname $CELERYD_PID_FILE`
277
    if [ ! -d "$CELERYD_LOG_DIR" ]; then
278
          mkdir -p $CELERYD_LOG_DIR
279
    fi
280
    if [ ! -d "$CELERYD_PID_DIR" ]; then
281
          mkdir -p $CELERYD_PID_DIR
282
    fi
283

    
284
    # Extra start-stop-daemon options, like user/group.
285
    if [ -n "$CELERYD_USER" ]; then
286
          DAEMON_OPTS="$DAEMON_OPTS --uid=$CELERYD_USER"
287
          chown "$CELERYD_USER" $CELERYD_LOG_DIR $CELERYD_PID_DIR
288
    fi
289
    if [ -n "$CELERYD_GROUP" ]; then
290
          DAEMON_OPTS="$DAEMON_OPTS --gid=$CELERYD_GROUP"
291
          chgrp "$CELERYD_GROUP" $CELERYD_LOG_DIR $CELERYD_PID_DIR
292
    fi
293

    
294
    if [ -n "$CELERYD_CHDIR" ]; then
295
          DAEMON_OPTS="$DAEMON_OPTS --workdir=\"$CELERYD_CHDIR\""
296
    fi
297

    
298

    
299
    check_dev_null() {
300
          if [ ! -c /dev/null ]; then
301
               echo "/dev/null is not a character device!"
302
               exit 1
303
          fi
304
    }
305

    
306

    
307
    export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"
308

    
309

    
310
    stop_workers () {
311
          $CELERYD_MULTI stop $CELERYD_NODES --pidfile="$CELERYD_PID_FILE"
312
    }
313

    
314

    
315
    start_workers () {
316
          $CELERYD_MULTI start $CELERYD_NODES $DAEMON_OPTS           \
317
                                        --pidfile="$CELERYD_PID_FILE"        \
318
                                        --logfile="$CELERYD_LOG_FILE"        \
319
                                        --loglevel="$CELERYD_LOG_LEVEL"     \
320
                                        --cmd="$CELERYD"                           \
321
                                        $CELERYD_OPTS
322
    }
323

    
324

    
325
    restart_workers () {
326
          $CELERYD_MULTI restart $CELERYD_NODES $DAEMON_OPTS        \
327
                                           --pidfile="$CELERYD_PID_FILE"     \
328
                                           --logfile="$CELERYD_LOG_FILE"     \
329
                                           --loglevel="$CELERYD_LOG_LEVEL"  \
330
                                           --cmd="$CELERYD"                        \
331
                                           $CELERYD_OPTS
332
    }
333

    
334

    
335

    
336
    case "$1" in
337
          start)
338
               check_dev_null
339
               start_workers
340
          ;;
341

    
342
          stop)
343
               check_dev_null
344
               stop_workers
345
          ;;
346

    
347
          reload|force-reload)
348
               echo "Use restart"
349
          ;;
350

    
351
          status)
352
               $CELERYCTL status $CELERYCTL_OPTS
353
          ;;
354

    
355
          restart)
356
               check_dev_null
357
               restart_workers
358
          ;;
359

    
360
          try-restart)
361
               check_dev_null
362
               restart_workers
363
          ;;
364

    
365
          *)
366
               echo "Usage: /etc/init.d/celeryd {start|stop|restart|try-restart|kill}"
367
               exit 1
368
          ;;
369
    esac
370

    
371
    exit 0
372

    
373
celeryd configuration
374
---------------------
375
celeryd requires a /etc/default/celeryd file to be in place.
376
Thus we are going to create this file (/etc/default/celeryd)::
377

    
378
    vim /etc/default/celeryd
379

    
380
Again if the directory conventions have been followed the file is (pay attention to the CELERYD_USER, CELERYD_GROUP and change accordingly)  ::
381

    
382
    # Default: false
383
    ENABLED="true"
384

    
385
    # Name of nodes to start, here we have a single node
386
    CELERYD_NODES="w1"
387
    # or we could have three nodes:
388
    #CELERYD_NODES="w1 w2 w3"
389

    
390
    # Where to chdir at start.
391
    CELERYD_CHDIR="/srv/flowspy"
392
    # How to call "manage.py celeryd_multi"
393
    CELERYD_MULTI="python $CELERYD_CHDIR/manage.py celeryd_multi"
394

    
395
    # How to call "manage.py celeryctl"
396
    CELERYCTL="python $CELERYD_CHDIR/manage.py celeryctl"
397

    
398
    # Extra arguments to celeryd
399
    #CELERYD_OPTS="--time-limit=300 --concurrency=8"
400
    CELERYD_OPTS="-E -B --schedule=/var/run/celery/celerybeat-schedule --concurrency=1 --soft-time-limit=180 --time-limit=1800"
401
    # Name of the celery config module.
402
    CELERY_CONFIG_MODULE="celeryconfig"
403

    
404
    # %n will be replaced with the nodename.
405
    CELERYD_LOG_FILE="/var/log/celery/fod_%n.log"
406
    CELERYD_PID_FILE="/var/run/celery/%n.pid"
407

    
408
    CELERYD_USER="root"
409
    CELERYD_GROUP="root"
410

    
411
    # Name of the projects settings module.
412
    export DJANGO_SETTINGS_MODULE="flowspy.settings"
413

    
414
Apache
415
------
416
Apache proxies gunicorn. Things are more flexible here as you may follow your own configuration and conventions. Create and edit /etc/apache2/sites-available/fod. You should set <server_name> and <admin_mail> along with your certificates. If under testing environment, you can use the provided snakeoil certs. If you do not intent to use Shibboleth delete or comment the corresponding configuration parts inside **Shibboleth configuration** ::
417

    
418
    vim /etc/apache2/sites-available/fod
419

    
420
Again if the directory conventions have been followed the file should be::
421

    
422
    <VirtualHost *:80>
423
        ServerAdmin webmaster@localhost
424
        ServerName  fod.example.com
425
        DocumentRoot /var/www
426

    
427
        ErrorLog ${APACHE_LOG_DIR}/fod_error.log
428

    
429
        # Possible values include: debug, info, notice, warn, error, crit,
430
        # alert, emerg.
431
        LogLevel debug
432

    
433
        CustomLog ${APACHE_LOG_DIR}/fod_access.log combined
434

    
435
        Alias /static       /srv/flowspy/static
436
          RewriteEngine On
437
          RewriteCond %{HTTPS} off
438
          RewriteRule ^/(.*) https://fod.example.com/$1 [L,R]
439
    </VirtualHost>
440

    
441
    <VirtualHost *:443>
442
        ServerName    fod.example.com
443
        ServerAdmin     webmaster@localhost
444
        ServerSignature        On
445

    
446
        SSLEngine on
447
        SSLCertificateFile    /etc/ssl/certs/fod.example.com.crt
448
        SSLCertificateChainFile /etc/ssl/certs/example-chain.pem
449
        SSLCertificateKeyFile    /etc/ssl/private/fod.example.com.key
450

    
451
        AddDefaultCharset UTF-8
452
        IndexOptions        +Charset=UTF-8
453

    
454
        ShibConfig       /etc/shibboleth/shibboleth2.xml
455
        Alias          /shibboleth-sp /usr/share/shibboleth
456

    
457

    
458
        <Location /login>
459
             AuthType shibboleth
460
             ShibRequireSession On
461
             ShibUseHeaders On
462
             ShibRequestSetting entityID https://idp.example.com/idp/shibboleth
463
             require valid-user
464
        </Location>
465

    
466
        # Shibboleth debugging CGI script
467
        ScriptAlias /shibboleth/test /usr/lib/cgi-bin/shibtest.cgi
468
        <Location /shibboleth/test>
469
             AuthType shibboleth
470
             ShibRequireSession On
471
             ShibUseHeaders On
472
             require valid-user
473
        </Location>
474

    
475
        <Location /Shibboleth.sso>
476
             SetHandler shib
477
        </Location>
478

    
479
        # Shibboleth SP configuration
480

    
481
        #SetEnv                       proxy-sendchunked
482

    
483
              <Proxy *>
484
               Order allow,deny
485
               Allow from all
486
               </Proxy>
487

    
488
               SSLProxyEngine           off
489
               ProxyErrorOverride     off
490
          ProxyTimeout     28800
491
             ProxyPass        /static !
492
             ProxyPass          /shibboleth !
493
             ProxyPass        /Shibboleth.sso !
494

    
495
               ProxyPass           / http://localhost:8081/ retry=0
496
               ProxyPassReverse / http://localhost:8081/
497

    
498
          Alias /static          /srv/flowspy/static
499

    
500
        LogLevel warn
501

    
502
        ErrorLog ${APACHE_LOG_DIR}/fod_error.log
503
          CustomLog ${APACHE_LOG_DIR}/fod_access.log combined
504

    
505
    </VirtualHost>
506

    
507
Now, enable your site. You might want to disable the default site if fod is the only site you host on your server::
508

    
509
    a2dissite default
510
    a2ensite fod
511

    
512
You are not far away from deploying FoD. When asked for a super user, create one::
513

    
514
    cd /srv/flowspy
515
    python manage.py syncdb
516
    python manage.py migrate longerusername
517
    python manage.py migrate flowspec
518
    python manage.py migrate djcelery
519
    python manage.py migrate accounts
520
    python manage.py migrate
521

    
522
If you have not changed the values of the PEER\_\*\_TABLE variables to False and thus you are going for a default installation (that is PEER\_\*\_TABLE variables are set to True) , then run::
523

    
524
    python manage.py migrate peers
525

    
526
If however you have set the PEER\_\*\_TABLE variables to False and by accident you have ran the command above, then you have to cleanup you database manually by dropping the peer\* tables plus the techc_email table. For MySQL the command is::
527

    
528
    DROP TABLE `peer`, `peer_networks`, `peer_range`, `peer_techc_emails`, techc_email;
529

    
530
Restart, gunicorn and apache::
531

    
532
    service gunicorn restart && service apache2 restart
533

    
534

    
535
Propagate the flatpages
536
=======================
537
Inside the initial_data/fixtures_manual.xml file we have placed 4 flatpages (2 for Greek, 2 for English) with Information and Terms of Service about the service.
538
To import the flatpages, run from root folder::
539

    
540
    python manage.py loaddata initial_data/fixtures_manual.xml
541

    
542

    
543

    
544
Testing the platform
545
====================
546
Log in to the admin interface via https:\/\/<hostname>\/admin. Go to Peer ranges and add a new range (part of/or a complete subnet), eg. 10.20.0.0/19
547
Go to Peers and add a new peer, eg. id: 1, name: Test, AS: 16503, tag: TEST and move the network you have created from Avalable to Chosen. From the admin front, go to User, and edit your user. From the bottom of the page, select the TEST peer and save.
548
Last but not least, modify as required the existing (example.com) Site instance (admin home->Sites). You are done. As you are logged-in via the admin, there is no need to go through Shibboleth at this time. Go to https:\/\/<hostname>\/ and create a new rule. Your rule should be applied on the flowspec capable device after aprox. 10 seconds. If no Shibboleth authentication is available, a https:\/\/<hostname>\/altlogin is provided.
549

    
550
Branding
551
========
552
Via the admin interface you can modify flatpages to suit your needs
553

    
554
Footer
555
------
556
Under the templates folder (templates), you can alter the footer.html file to include your own footer messages, badges, etc.
557

    
558
Welcome Page
559
------------
560
Under the templates folder (templates), you can alter the welcome page - welcome.html with your own images, carousel, videos, etc.