Tips to keep your Django/mod_python memory usage down
Updated Jan 28 at 04:44 CDT (first posted May 30 at 09:57 CDT) by Remi in Django, Memory, Tips - 16 comment(s)
Most people manage to run their Django site on mod_python within the memory limits of their "Shared 1" or "Shared 2" plans but a few people are struggling to stay within the limits.
So here are a few tips that you can use to try and keep your memory usage down when using Django on mod_python:
- Make sure that you set DEBUG to False in settings.py: if it isn't, set it to False and restart apache. Amongst other things, DEBUG mode stores all SQL queries in memory so your memory usage will quickly increase if you don't turn it off.
- Use "ServerLimit" in your apache config: by default apache will spawn lots of processes, which will use lots of memory. You can use the "ServerLimit" directive to limit these processes. A value of 3 or 4 is usually enough for most sites if your static data is not served by your Django instance (see below).
- Check that no big objects are being loaded in memory: for instance, check that your code isn't loading hundreds or thousands of database records in memory all at once. Also, if your application lets people download or upload big files, check that these big files are not being loaded in memory all at once.
- Serve your static data from our main server: this is a general advice for all django sites: make sure that your static data (images, stylesheets, ...) is served directly by our main apache server. This will save your Django app from having to serve all these little extra requests. Details on how to do that can be found here and here.
- Use "MaxRequestsPerChild" in your apache config: sometimes there are some slow memory leaks that you can't do anything about (they can be in the tools that you use themselves for instance). If this is the case then you can use the "MaxRequestsPerChild" to tell apache to only serve a certain number of requests before killing the process and starting a fresh one. Reasonable values are usually between 100 and 1000. Another more extreme/uglier version of this technique is to setup a cronjob to run "stop/start" once in a while.
- Find out and understand how much memory you're using: to find out what your processes are and how much memory they're using, you can run the "ps -u <username> -o pid,rss,command" command, like this: As you can see we have three "httpd" processes running that use respectively 3848KB, 10312KB and 9804KB of memory (there are various ways to interpret the memory used by a process on Linux and we have chosen to use the "Resident Set Size" (RSS) or your processes). The first one is the apache "supervisor" and the other two are the "workers" (in this example, "ServerLimit" is set to 2). The memory used by the supervisor usually doesn't change too much, but the memory used by the workers can increase greatly if you have bad memory leaks in your application. So the total memory used by our Apache/django instance in this example is 3848KB + 10312KB + 9804KB = 23MB.
16 comments:
Arthur Debert said on 2007-06-01 10:36:22:
Thanks for the tips.A quick heads up on file uploads. Django stores all file uploads in memory[1]. This means that if you allow users to upload files, your memory usage will skyrocket.
Not only it keeps files in memory, but it will keep 2 to 3 times the file's size. Thus, for uploads that aren't tiny (such as a 50k jpegs) you will probably have to bypass django on the upload all together (this is harder to do using the admin).
This has been a long standing issue, but it's not a trivial fix. I've tried hacking it once, but no dice.
Arthur
http://code.djangoproject.com/ticket/2070
OMG Thank You!!! said on 2007-06-14 18:36:05:
Our server was periodically going crazy and jumping up over 100 load average for minutes at a time causing the whole site to be unresponsive. ServerLimit was not set, I knew that 40 servers worked fine, so I set it to that. MaxRequestsPerChild was 0, so I set it to 500. Now it's working like a charm!!# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 30
was
# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 0
I have had three different techs trying to solve this for 4 days. You page fixed it in one restart. I can't thank you enough for taking the time to post this information.
Ryan said on 2007-06-28 06:19:06:
Thank you very much for listing these tips. For someone like me coming from a PHP background, I wouldn't have known where to start but this guide makes it straight forward and effective!
Pablo Rosales said on 2007-07-26 11:22:26:
Thanks for the tips! I'll be hosting one Django site here if all works well in a few months, a dedicated server will be the best solution.Regards, Pablo
Mike/ZodLogic said on 2007-11-07 01:13:59:
Not only does the Apache soft 'restart' command not necessarily free up all memory, I found that it was actually causing a memory leak-like effect in the parent/master Apache process. At least for my app and my config. Every time I issued 'restart' my master Apache process gained 600 kbytes in RSS. Every time, with nothing else happening. I had been doing many restarts like this while doing dev hacking recently and was bitten by this. A workaround is to write yourself a "hard" restart script that just calls stop, then start. (Probably add a "sleep 1" call in between to reduce chance for errors due to races on the process lifecyle.) One diff between a hard and soft restart like this is that in the hard restart the master process is blown away, and for me, that's where the leak effect was happening.
bluszcz said on 2007-12-06 05:30:10:
The best tip - do not use mod_python.
wordview said on 2008-04-12 17:32:55:
these might be useful:alias memcheck='ps -u $LOGNAME -o pid,rss,command'
alias memtotal='ps -u $LOGNAME h -o rss | (tr '\n' +; echo 0) | bc'
oncero said on 2008-06-13 11:08:04:
another version of the alias, calc the total mem used by all https procs:
alias memhttpd="ps -u $LOGNAME -o pid,rss,command | grep httpd | grep -v grep | awk 'BEGIN {sum=0} {sum=sum+\$2} END {printf(\"%f MB\n\",sum/1024)}'"
output looks like this:
33.042969 MB
7times9 said on 2008-06-15 14:24:59:
you have unescaped quotes inside quotes, so you need:alias memtotal="ps -u $LOGNAME h -o rss | (tr '\n' +; echo 0) | bc"
Brandon Konkle said on 2008-07-16 12:25:18:
To Arthur Debert: Over a year later, the bug you mentioned in your post above has finally been fixed! As of July 1st '08 - http://www.djangoproject.com/documentation/upload_handling/#where-uploaded-data-is-stored
Lucas Hazel said on 2008-10-07 03:02:30:
If you are using apache it also helps to turn off all modules you aren't using.If you are using a prefork configuration this will save a lot of memory.
Steve Losh said on 2009-01-14 07:57:51:
Thanks for posting this information; it's really helpful and was written very well.@bluszcz:
What alternative would you recommend? mod_wsgi is an obvious one, but even their official site says that its performance isn't going to be noticeably better than mod_python when using a framework like Django: http://code.google.com/p/modwsgi/wiki/PerformanceEstimates
Ben Keating said on 2009-04-30 18:32:24:
How about Django/Python code practices to keep memory usage tight? Any suggestions on best practices?Should I be clearing my views with an exit or break or? What other tricks do you keep in mind when developing to keep your footprint as small as possible?
Sean said on 2009-05-01 10:17:11:
@Ben - re exit/break I don't believe you need to do anything like that in your views. What you do want to do is ensure that your view is not returning *huge* recordsets or otherwise dealing with huge amounts of data.Other than that, just remember that Django is Python - if you write efficient Python code, then you'll have an efficient Django app :)
rick said on 2009-06-11 03:06:21:
Are there plans to display memory usage in the admin panel, because it's sort of inconvenient to run the command and do the math...
@rick - It's on our to-do list :)