Memory usage
Posted Jul 26 at 11:30 CDT by Remi in Help, Memory - 11 comment(s)
Introduction
Our plans come with the ability to have SSH access to our servers and to run long-running processes.
This greatly facilitates the ability to run and debug powerful tools that you might want to use for your site, but this comes at a cost.
Since you're sharing the machine with other people we have to make sure that your processes don't use too much resources. The resources that your processes use include:
- Memory
- CPU
- Disk IO
- Network IO
In this article we'll focus on memory usage.
How much memory do my processes use ?
There are various ways to interpret the memory used by a process on Linux. We have chosen to use the "Resident Set Size" (RSS) or your processes.
You can easily check how much memory your processes use by typing the following command:
The second column is the memory used by your process, in KB.
So, in this example you can see that user "rdelon" uses: 128584 + 200 + 1256 = 130040 KB = 126 MB of memory (this includes the "ps" process itself)
How can I keep my memory usage down ?
This is a difficult question... The memory your processes use can be broken down in two categories:
- the "base memory usage" of the tool you're using (ex: 50MB for Zope, 100MB for Zope+Plone, ...)
- the additional memory used by your actual application
You have very little control over the base memory usage of the tool you're using (other than switching to another tool or submitting patches to the maintainer to reduce memory usage).
But you can be careful about the additional memory used by your application.
Here are a few tips to keep that memory usage low:
- use less thread/processes: if you use a tool that starts multiple threads or processes to handle concurrent requests, try to configure the number of threads and processes to a minimum.
- watch for memory leaks: this is the #1 problem causing memory to grow. To check if you might have a memory leak you can use tools like Apache's "ab" to simulate lots of requests to your website and see if the memory of your process leaks.
- watch what you keep in memory: sometimes you want to store some data in memory for performance reason but there is a tradeoff to find between memory usage and performance. Only store in memory data that is accessed very often or is very costly to retrieve.
- clean up whatever you can: for instance, if you store session data in memory, make sure you clean up that data when it is no longer needed. Some tools also require some maintenance tasks to be performed regularly (for instance, with Zope, you should pack your Data.fs file regularly).
- use tools to help you: Most "modern" languages such as Python and Ruby provide ways to inspect your running program and see where the memory might be used. You could for instance provide a URL (password-protected of course) that returns all kinds of information about your running program. That there are also some advanced tools (for most languages) to keep track of the memory that's being allocated/de-allocated in your program.
What happens if I use more memory than I'm allowed ?
If your processes use more memory than allowed we will first send you an e-mail about it and we'll try to help you fix the problem. If they use *way* more than allowed (to the point that it impacts other processes on the server) we will restart your processes.
If the problem persists you will have to upgrade your account to a plan that allows more memory.
11 comments:
Robert said on 2006-10-06 23:08:57:
1.I don't understand "Max total RAM used by long-running processes". Is this maximum the sum of ALL long-running processes? Or just one individual process (and you can have more than one)?
2.
So does that limit me only to Shared 4 if I want to run a plone site?
3. If I'm running ZEO, would that affect the memory usage? I'll be using the other zope instance for debugging only If I'm creating extra plone sites from the same instance, does that affect memory?
4. is database memory usage (postgresql) or apache included in the total?
Christian Wilcox said on 2006-10-10 07:55:20:
Here's a useful modification to the command listed above:ps -u username -o pid,rss,command | awk '{print $0}{sum+=$2} END {print "Total", sum}'
This will display a list of processes followed by total memory usage.
suresh said on 2008-02-21 11:14:13:
can you average the usage, so I am allowed higher usages on an infrequent basis?demos and such
juliux said on 2008-02-21 17:55:03:
Guys If some Open Source Applications uses too much memory such as zope for example, and the client bought a "economic" plan...why you do not disable this apps in the apps menu in the pannel? I am sure, many clients like me had changed the plan in the beginning.
Thanks
Remi said on 2008-02-22 16:37:13:
suresh:"can you average the usage, so I am allowed higher usages on an infrequent basis?": sorry, we can't do that otherwise if several people go over their limit at the same time the machine could run out of memory.
juliux: good idea, we'll look into doing this (note that we're already checking that when you signup).
Remi.
chris said on 2008-04-06 10:03:07:
I would have to agree with Juliux, when I purchased my plan, specifically for Zope and Plone I expected that these applications would run fine and be within the memory limits.Quote from your shared hosting plans page.
"From static HTML, PHP and CGI to Rails, Django, Turbogears and Plone, these plans let you run any software you want. You can even install your own custom software in your home directory."
As I purchased a Shared 3 plan (140mb memory) and not a "Discount" plan. I am running a handfull (9) of sites and the last "message" I got from webfaction stated that my memory usage is now up to a whopping 240MB.
How can this be?
Sean Fulmer said on 2008-04-07 09:41:05:
Hi Chris - "any software you want" is not the same thing as "as much software as you'd like" :)Plone has a very high memory requirement, so there are, of course, limits to the number of Plone sites you can run on any hosting plan.
--
Sean
Wayne Glover said on 2008-04-08 15:42:52:
hi, in the control panel under 'usage', the memory usage isn't listed.is it possible to include this so we could have a better idea of our memory usage?
i realize their are command lines, but i'd rather not have to resort to, well thinking!
thanks
Wayne Glover said on 2008-04-08 15:45:26:
forgot to ask:why does the memory increase with time?
I have very little content in my plone site and have not added much lately.
Is this a situation where a restart would decrease the memory usage?
thanks
David Sissitka said on 2008-04-09 02:32:34:
> is it possible to include this so we could have a better idea of our memory usage?I suggest posting your request in our forum, if enough people would like to see it implemented... :)
> i realize their are command lines, but i'd rather not have to resort to, well thinking!
http://forum.webfaction.com/viewtopic.php?id=1263 should be of some assistance to you, no thought required. :)
> why does the memory increase with time?
There are a number of possible reasons. I'm not a Plone/Zope expert but I believe that the two most common causes are:
1) Not packing your Data.fs regularly. See:
https://help.webfaction.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=48
2) The first time you visit a page after starting Plone/Zope it's cached in memory. So, Plone/Zope's memory usage right after you restart or start it will increase until every page has been visited.
I'll make sure that someone that's more familiar with Plone/Zope than me sees this.
>1) Not packing your Data.fs regularly. See:
>
>https://help.webfaction.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=48
Theoretically, not packing your ZODB should generally not impact memory usage. Anecdotally, it does - and of course, a well-maintained site will generally perform better than an ill-maintained site.
>2) The first time you visit a page after starting Plone/Zope it's cached in memory. So, Plone/Zope's memory usage right after you restart or start it will increase until every page has been visited.
Not quite - Zope's various caching mechanisms (and there are a variety of them!) generally have a threshold for the number of entries, a setting for the maximum age of an entry, and a cleanup interval at which the first two settings are checked and applied.
The entry threshold setting isn't a hard limit... it's just a trigger. So, you might have a threshold of 1000, but suddenly get a ton of traffic and end up with 10k objects in the cache. This will trigger a cleanup action, but while those objects are 10k there, your memory usage will be high.
So - if your memory usage keeps going up and up and up, check the cache settings in the ZMI.. for Plone this would be the RAMCache, HTTPCache, and RSSCache objects. It can be a lot of trial and error to properly tune the caching for a Zope site, but it can be done :)