Setting up the reddit app without root access

We manage the servers for you which means that you don’t have to worry about setting them up and most importantly maintaining them. We provide a team of systems administrators who work around the clock to do things like applying security patches, monitoring servers, fixing server issues when they arise, building RPMs with new versions of tools and backing up your data.
For this to work smoothly we have to make sure that our servers are set up in a certain way (with some standard linux tools and some of our own tools) and it means that you don’t get root access to the servers. In practice this is rarely an issue because you can usually install anything you need in your HOME directory, even when the tools’ installation guides assume that you do have root access.

As an example we’re going to set up the reddit app on our servers. This is the code running the popular reddit.com website. Here is the app running on a WebFaction server:

reddit app on WebFaction

The list of dependencies is a bit daunting and two of the tools (the reddit app itself and Cassandra) assume that you have root access in their installation guides. Let’s follow the reddit installation guide and see how we can work around the lack of root access. There are a lot of steps to install the app but we’ll be rewarded in the end by a fast and nice-looking reddit app instance:

1. Create a “custom app” in the WebFaction control panel

The reddit app runs the Pylons web server so we go to “Domains / websites -> Applications”” in the control panel and create a “Custom application running on port”. Let’s call it “reddit”. All this does is create an empty directory names $HOME/webapps/reddit on the server and give us a port that we can use to run the app on (in this case the port is 17594). When we point a domain to that app the front Nginx server on the server will get configured to proxy all requests to that port.

2. Create a website in the control panel to point a domain to the custom app

In step 1 we created a custom app called “reddit”. The next step is to go to “Domains / websites -> Websites” in the control panel and to create a website. Let’s set it up to serve the “reddit” app at http://demo.webfactional.com.

3. Create a Postgresql database from the control panel and load the reddit SQL functions

The reddit app requires a Postgresql database. To create one we simply go to “Databases” in the control panel and create a new Postgresql database called “demo_reddit”. We take note of the password and then SSH into the server as our normal user (“demo” in this example) to load the reddit SQL functions:

[demo@web356 reddit]$ cd $HOME/webapps/reddit/reddit
[demo@web356 reddit]$ psql -U demo_reddit demo_reddit < sql/functions.sql
Password for user demo_reddit:
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE VIEW
[demo@web356 reddit]$

4. Download the reddit app code and update the config files

[demo@web356 ~]$ cd $HOME/webapps/reddit
[demo@web356 reddit]$ git clone git://github.com/reddit/reddit.git
Cloning into reddit...
remote: Counting objects: 21254, done.
remote: Compressing objects: 100% (4888/4888), done.
remote: Total 21254 (delta 17314), reused 20069 (delta 16305)
Receiving objects: 100% (21254/21254), 14.36 MiB | 3.63 MiB/s, done.
Resolving deltas: 100% (17314/17314), done.
[demo@web356 reddit]$
# Update the config file with our own settings
[demo@web356 ~]$ sed -i 's/reddit.local/demo.webfactional.com/g' $HOME/webapps/reddit/reddit/r2/example.ini
[demo@web356 ~]$ sed -i 's/db_user = reddit/db_user = demo_reddit/' $HOME/webapps/reddit/reddit/r2/example.ini
[demo@web356 ~]$ sed -i 's/db_pass = password/db_pass = secret-password/' $HOME/webapps/reddit/reddit/r2/example.ini
[demo@web356 ~]$ sed -i 's/reddit,   127.0.0.1/demo_reddit,   127.0.0.1/' $HOME/webapps/reddit/reddit/r2/example.ini

5. Setup SunJDK

The reddit app requires Cassandra which runs on Java. All WebFaction servers come with OpenJDK but in our tests Cassandra didn't seem to work with that version of Java. So we're going to install SunJDK in our HOME directory and point our JAVA_HOME environment variable to it:

 <download jdk-7u7-linux-x64.tar.gz from http://www.oracle.com/technetwork/java/javase/downloads/jdk7u7-downloads-1836413.html and place the file in $HOME on the server>
[demo@web356 ~]$ cd $HOME
[demo@web356 ~]$ tar xf jdk-7u7-linux-x64.tar.gz
[demo@web356 ~]$ export JAVA_HOME=$HOME/jdk1.7.0_07

6. Setup Cassandra

[demo@web356 ~]$ cd $HOME
[demo@web356 ~]$ wget http://apache.mesi.com.ar/cassandra/1.1.5/
        apache-cassandra-1.1.5-bin.tar.gz
[...]
2012-10-15 15:56:27 (3.97 MB/s) - apache-cassandra-1.1.5-bin.tar.gz
[demo@web356 ~]$ tar xf apache-cassandra-1.1.5-bin.tar.gz
# Tell Cassandra to put its data in $HOME/webapps/reddit/cassandra-data
[demo@web356 ~]$ mkdir $HOME/webapps/reddit/cassandra-data
[demo@web356 ~]$ sed -i 's/\/var\/lib\/cassandra\/data/\/home\/demo\/webapps\/reddit\/cassandra-data/' $HOME/apache-cassandra-1.1.5/conf/cassandra.yaml
[demo@web356 ~]$ sed -i 's/\/var\/lib\/cassandra\/data/\/home\/demo\/webapps\/reddit\/cassandra-data/' $HOME/apache-cassandra-1.1.5/conf/cassandra.yaml
[demo@web356 ~]$ sed -i 's/\/var\/lib\/cassandra\/commitlog/\/home\/demo\/webapps\/reddit\/cassandra-data\/commitlog/' $HOME/apache-cassandra-1.1.5/conf/cassandra.yaml
[demo@web356 ~]$ sed -i 's/\/var\/lib\/cassandra\/saved_caches/\/home\/demo\/webapps\/reddit\/cassandra-data\/saved_caches/' $HOME/apache-cassandra-1.1.5/conf/cassandra.yaml
# Start Cassandra
[demo@web356 ~]$ cd $HOME/apache-cassandra-1.1.5/bin
[demo@web356 bin]$ ./cassandra
xss =  -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms3980M -Xmx3980M -Xmn800M -XX:+HeapDumpOnOutOfMemoryError -Xss180k
[...]
 INFO 16:12:00,714 Node localhost/127.0.0.1 state jump to normal
 INFO 16:12:00,714 Bootstrap/Replace/Move completed! Now serving reads.

Cassandra is now running. In a production setup we would have started Cassandra in the background but in this example we started it in the foreground so we'll leave that window open and will run the next commands in a new SSH session in another window:

[demo@web356 ~]$ cd /home/demo/apache-cassandra-1.1.5/bin
[demo@web356 bin]$ ./cassandra-cli -h localhost
Connected to: "Test Cluster" on localhost/9160
Welcome to Cassandra CLI version 1.1.5

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown] create keyspace reddit;
9a890e19-4fe7-3655-afdb-95bcc7065947
Waiting for schema agreement...
... schemas agree across the cluster
[default@unknown] use reddit;
Authenticated to keyspace: reddit
[default@reddit] create column family permacache with column_type = 'Standard' and comparator = 'BytesType';
5f39dc3d-e8fc-34db-bb59-f3806851300c
Waiting for schema agreement...
... schemas agree across the cluster
[default@reddit]

Note that depending on its configuration Cassandra can use a lot of memory (1GB or more) so you will need enough memory on your account to run it.

7. Install Erlang

The reddit app requires RabbitMQ, which in turn requires Erlang. Since Erlang isn't installed globally on the WebFaction servers we'll just install it in our HOME directory:

[demo@web356 ~]$ cd
[demo@web356 ~]$ wget http://www.erlang.org/download/otp_src_R15B02.tar.gz
[...]
2012-10-16 11:18:19 (430 KB/s) - otp_src_R15B02.tar.gz
[demo@web356 ~]$ tar xf otp_src_R15B02.tar.gz
[demo@web356 ~]$ cd otp_src_R15B02
[demo@web356 otp_src_R15B02]$ mkdir $HOME/erlang
[demo@web356 otp_src_R15B02]$ ./configure --prefix=$HOME/erlang
Ignoring the --cache-file argument since it can cause the system to be erroneously configured
Disabling caching
checking build system type... x86_64-unknown-linux-gnu
[...]
                 Using fakefop to generate placeholder PDF files.

*********************************************************************
[demo@web356 otp_src_R15B02]$ make
cd erts && ERL_TOP=/home/demo/otp_src_R15B02 make NO_START_SCRIPTS=true  FLAVOR=
make[1]: Entering directory `/home/demo/otp_src_R15B02/erts'
[...]
make[2]: Leaving directory `/home/demo/otp_src_R15B02/erts/start_scripts'
make[1]: Leaving directory `/home/demo/otp_src_R15B02/erts'
[demo@web356 otp_src_R15B02]$ make install
[...]
cd erts && ERL_TOP=/home/demo/otp_src_R15B02 make NO_START_SCRIPTS=true  FLAVOR=
make[1]: Entering directory `/home/demo/otp_src_R15B02/erts'
ln -s ../lib/erlang/bin/ct_run ct_run
ln -s ../lib/erlang/bin/run_test run_test
[demo@web356 otp_src_R15B02]$ export PATH=$PATH:$HOME/erlang/bin

8. Setup RabbitMQ

[demo@web356 ~]$ cd
[demo@web356 ~]$ wget http://www.rabbitmq.com/releases/rabbitmq-server/v2.8.7/rabbitmq-server-generic-unix-2.8.7.tar.gz
[...]
2012-10-16 13:41:53 (686 KB/s) - rabbitmq-server-generic-unix-2.8.7.tar.gz
[demo@web356 ~]$ tar xf rabbitmq-server-generic-unix-2.8.7.tar.gz
# Set a few environment variables to tell RabbitMQ to only listen to localhost
[demo@web356 ~]$ export RABBITMQ_NODENAME=rabbit@localhost
[demo@web356 ~]$ export RABBITMQ_NODE_IP_ADDRESS=127.0.0.1
[demo@web356 ~]$ export ERL_EPMD_ADDRESS=127.0.0.1
[demo@web356 ~]$ cd rabbitmq_server-2.8.7/sbin
[demo@web356 sbin]$ ./rabbitmq-server
Activating RabbitMQ plugins ...
[...]
starting notify cluster nodes                                         ...done
broker running

The RabbitMQ server is now running. Again, in a production setup we would have started it in the background but in this example we started it in the foreground so we'll leave that window open and will run the next commands in a new SSH session in another window.
The next step is to add a user in RabbitMQ:

[demo@web356 ~]$ export PATH=$PATH:/home/demo/erlang/bin
[demo@web356 ~]$ export RABBITMQ_NODENAME=rabbit@localhost
[demo@web356 ~]$ export RABBITMQ_NODE_IP_ADDRESS=127.0.0.1
[demo@web356 ~]$ export ERL_EPMD_ADDRESS=127.0.0.1
[demo@web356 ~]$ cd $HOME/rabbitmq_server-2.8.7/sbin
[demo@web356 sbin]$ ./rabbitmqctl add_user reddit reddit
Creating user "reddit" ...
...done.
[demo@web356 sbin]$ ./rabbitmqctl set_permissions -p / reddit ".*" ".*" ".*"
Setting permissions for user "reddit" in vhost "/" ...
...done.
[demo@web356 sbin]$

9. Run memcache

Memcache is already install on all WebFaction server so we just need to start an instance:

[demo@web356 ~]$ memcached

The Memcache server is now running. Since we started it in the foreground we'll leave that window open and will run the next commands in a new SSH session in another window.

10. Install Cython

[demo@web356 ~]$ cd
[demo@web356 ~]$ wget http://cython.org/release/Cython-0.17.1.tar.gz
[...]
2012-10-16 14:13:01 (2.36 MB/s) - Cython-0.17.1.tar.gz
[demo@web356 ~]$ tar xf Cython-0.17.1.tar.gz
[demo@web356 ~]$ cd Cython-0.17.1
[demo@web356 Cython-0.17.1]$ python2.7 setup.py install
Compiling module Cython.Plex.Scanners ...
[...]
Writing /home/demo/lib/python2.7/Cython-0.17.1-py2.7.egg-info
[demo@web356 Cython-0.17.1]$

11. Install PyCaptcha

[demo@web356 ~]$ cd
[demo@web356 ~]$ svn co http://svn.navi.cx/misc/trunk/pycaptcha/
A    pycaptcha/BUGS
[...]
 U   pycaptcha
Checked out revision 12529.
[demo@web356 ~]$ cd pycaptcha/
[demo@web356 pycaptcha]$ python2.7 setup.py install
running install
running build
[...]
Writing /home/demo/lib/python2.7/PyCAPTCHA-0.4-py2.7.egg-info
[demo@web356 pycaptcha]$

12. Install libmemcached-0.53 and pylibmc

The reddit app uses pylibmc which requires libmemcached-0.32 or later. Unfortunately the libmemcached version installed globally on the WebFaction servers is 0.31. So we're going to install a newer version (0.53) in our HOME directory and then build pylibmc against this newer version:

[demo@web356 ~]$ cd
[demo@web356 ~]$ wget https://launchpad.net/libmemcached/1.0/0.53/+download/libmemcached-0.53.tar.gz
[...]
2012-10-16 14:30:12 (322 KB/s) - libmemcached-0.53.tar.gz
[demo@web356 ~]$ tar xf libmemcached-0.53.tar.gz
[demo@web356 ~]$ cd libmemcached-0.53
[demo@web356 libmemcached-0.53]$ mkdir $HOME/libmemcached
[demo@web356 libmemcached-0.53]$ ./configure --prefix=$HOME/libmemcached
[...]
   * Warnings as failure:       no
---
[demo@web356 libmemcached-0.53]$ make install
[...]
make[1]: Leaving directory `/home/demo/libmemcached-0.53'
[demo@web356 libmemcached-0.53]$ export LD_LIBRARY_PATH=$HOME/libmemcached/lib
[demo@web356 libmemcached-0.53]$ cd
[demo@web356 ~]$ wget http://pypi.python.org/packages/source/p/pylibmc/pylibmc-1.2.3.tar.gz
[...]
2012-10-16 14:48:05 (390 KB/s) - pylibmc-1.2.3.tar.gz
[demo@web356 ~]$ tar xf pylibmc-1.2.3.tar.gz
[demo@web356 ~]$ cd pylibmc-1.2.3
[demo@web356 pylibmc-1.2.3]$ python2.7 --with-libmemcached=/home/demo/libmemcached/ setup.py install
running install
[..]
Writing /home/demo/lib/python2.7/pylibmc-1.2.3-py2.7.egg-info
[demo@web356 pylibmc-1.2.3]$ 

13. Install Pylons

[demo@web356 ~]$ cd
[demo@web356 ~]$ wget http://pypi.python.org/packages/source/P/Pylons/Pylons-1.0.1.tar.gz
[...]
2012-10-16 14:06:50 (2.40 MB/s) - Pylons-1.0.1.tar.gz
[demo@web356 ~]$ tar xf Pylons-1.0.1.tar.gz
[demo@web356 ~]$ cd Pylons-1.0.1
[demo@web356 Pylons-1.0.1]$ python2.7 setup.py install
[...]
Installed /home/demo/lib/python2.7/repoze.lru-0.6-py2.7.egg
Finished processing dependencies for Pylons==1.0.1
[demo@web356 Pylons-1.0.1]$

14. Build the reddit app

[demo@web356 ~]$ cd $HOME/webapps/reddit/reddit/r2
[demo@web356 r2]$ sed -i 's/PYTHON=python/PYTHON=python2.7/' Makefile
[demo@web356 r2]$ make pyx
[+] including definitions from Makefile.py
cython r2/lib/utils/_utils.pyx
[...]
gcc -pthread -shared build/temp.linux-x86_64-2.7/r2/lib/c/filters.o -L/usr/local/lib -lpython2.7 -o /home/demo/webapps/reddit/reddit/r2/Cfilters.so
touch build/pyx-buildstamp
[demo@web356 r2]$ python2.7 setup.py develop
[...]
Using /home/demo/lib/python2.7/MarkupSafe-0.15-py2.7-linux-x86_64.egg
Finished processing dependencies for r2==0.0.0dev
[demo@web356 r2]$

15. Start the reddit app

We have now installed all the dependencies and configured the reddit app with our settings. The final step is to start the reddit app on the port that was given to us in step 1:

[demo@web356 ~] cd $HOME/webapps/reddit/reddit/r2
[demo@web356 r2]$ paster serve --reload example.ini http_port=17594
Starting subprocess with file monitor
Overriding g.http_port to 17594
reddit app web356.webfaction.com:11612 started a84ad12 at 2012-10-16 15:00:28.862558
Starting server in PID 11612.
serving on 0.0.0.0:17594 view at http://127.0.0.1:17594

16. Check out our handy work

We point our browser at http://demo.webfactional.com/ and can finally get our reward:
reddit app on WebFaction

17. A bit of fun

We couldn't resist doing some basic load-testing on the app to see what kind of figures we'd get:

[demo@web356 ~]$ ab -n 1000 http://demo.webfactional.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking demo.webfactional.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx
Server Hostname:        demo.webfactional.com
Server Port:            80

Document Path:          /
Document Length:        30282 bytes

Concurrency Level:      1
Time taken for tests:   3.349 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      30607000 bytes
HTML transferred:       30282000 bytes
Requests per second:    298.63 [#/sec] (mean)
Time per request:       3.349 [ms] (mean)
Time per request:       3.349 [ms] (mean, across all concurrent requests)
Transfer rate:          8926.04 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     3    3   1.0      3      35
Waiting:        3    3   1.0      3      34
Total:          3    3   1.0      3      35

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      3
  75%      3
  80%      3
  90%      4
  95%      4
  98%      4
  99%      4
 100%     35 (longest request)
[demo@web356 ~]$

Almost 300 requests per second on a simple shared account and without any tweaking. That's a pretty good figure.

Final notes

The steps to set up an app such as the reddit app on a WebFaction shared server without root access are almost the same as the steps with root access. A few things are slightly harder (having to specify the --prefix=$HOME/tool option when compiling certain tools) but on the other hand you won't have to do all the sysadmin work that our team does: applying security patches to globally installed software, monitoring the server, fixing server issues when they arise etc.

This entry was posted in General. Bookmark the permalink.

7 Responses to Setting up the reddit app without root access

  1. Rommel says:

    WebFaction always going above and beyond. They are now reading my mind and blogging about things I need before I even ask for them.

  2. Jaya says:

    How is the tcp port allocated for cassandra in shared server?

    • Remi D says:

      Create a “Custom application listening on port” in the control panel and this will give you a port number which you can use for your Cassandra server.

      • Jaya says:

        @Remi thanks for the clarification. Was curious that the article did not mention it and was troubling my thought that could it be a CGroup feature I missed(sandboxed tcp port).

  3. Rommel says:

    By the way, a minor gripe:
    Step 3 is asking you to load the SQL functions, but you don’t download them until Step 4. You might want to do the reddit cloning before Step 3 unless you have magic sql-appearing powers. I envy your sql-appearing powers.

  4. fgallaire says:

    Step 4:
    sed -i ‘s/reddit, 127.0.0.1/demo_reddit, 127.0.0.1/’ $HOME/webapps/reddit/reddit/r2/example.ini
    should be
    sed -i ‘s/reddit, 127.0.0.1/demo_reddit, 127.0.0.1/g’ $HOME/webapps/reddit/reddit/r2/example.ini

    Step 5:
    On the Centos 5 servers, we need the linux-i586version.

    Step 6:
    The download link is dead and
    sed -i ‘s/\/var\/lib\/cassandra\/data/\/home\/demo\/webapps\/reddit\/cassandra-data/’ $HOME/apache-cassandra-1.1.5/conf/cassandra.yaml
    is written twice.
    On Centos 5 servers, cassandra refuse to start :
    Uncomment and set MAX_HEAP_SIZE=”2G” and HEAP_NEWSIZE=”800M” in apache-cassandra-1.1.5/conf/cassandra-env.sh

  5. fgallaire says:

    Step 12:
    export LD_LIBRARY_PATH=$HOME/libmemcached/lib
    should be
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/libmemcached/lib

    python2.7 –with-libmemcached=/home/demo/libmemcached/ setup.py install
    running install
    should be
    python2.7 setup.py install running install –with-libmemcached=/home/demo/libmemcached/

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>