apc futex_wait lockdown make your apache freeze over

We had the typical LAMP setup going on, with Drupal as the base CMS and APC for bytecode cache. We needed a good caching engine so I figured why not use APC's user cache. Well, we tried the APC Cache Drupal Module which, with minor fixes proved to work very nicely. That is, until we actually put this all thing on production.

The first thing we had was having our apache hang and not respond to any user requests. We susspected network issues, especially since netstat -na showed that all the apache processes were hanging on SYN_WAIT. However, since apache restart solved the issue i started to suspect this was something else.

To make a long (very long) story short, I got strace on our prod machines to find out that apache was either hanging on futex_lock(....FUTEXT_WAIT...) or doing infinte loops on the same functions.

To make even a longer story short, I got gdb installed on those machines and the backtrace clearly indicated that the locks were from APC user-cache calls.

We decided to abandon APC user-cache and switch to memcached which proved faster and had less lockdowns.

The funny thing is that when we talked about this over dinner the same evening a developer from another team just pointed me to this article by one of the APC leaders: (or something) How to Dismantle an APC Bomb which has been around for over a year. I am supprised and shocked that such a information is hidded so well and not mentioned anywere in the docs. Moreover, I went through the APC code again after reading this post (I went through it once when i started analysing the problem) and it seems that this is not even close to being resolved. there are no patches and no TODOs and nothing of the sort. From reading the code the entire user-cache needs a major re-write. What gives?

(this is a post i wrote a couple of months ago, never had time to finish it. Unfortunately this is still not fixed afaik)
EDIT (07/2009): http://pecl.php.net/bugs/bug.php?id=15179 reports this to be fixed. If anyone can confirm this please send me a note so that I could update this post

slow lstat, slow php, slow drupal

If you are short on time go to bottom line.

Ok, the lstat() and stat() system calls system call are not really slow per se. But take a look at an apache strace -r dump from a drupal installation I was testing:

[user@host ~]# strace -o strace.load -r -s 256 -p <PID>
...
0.000081 getcwd("/mnt/var/www/html/drupal", 4096) = 32
0.000048 lstat("/mnt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.000805 lstat("/mnt/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.001596 lstat("/mnt/var/www", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.000104 lstat("/mnt/var/www/html", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
0.000362 lstat("/mnt/var/www/html/drupal", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0
0.000804 lstat("/mnt/var/www/html/drupal/sites", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.057575 lstat("/mnt/var/www/html/drupal/sites/all", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.000297 lstat("/mnt/var/www/html/drupal/sites/all/modules", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.014762 lstat("/mnt/var/www/html/drupal/sites/all/modules/Internet", {st_mode=S_IFDIR|0755, st_size=4096,
0.000214 lstat("/mnt/var/www/html/drupal/sites/all/modules/Internet/date", {st_mode=S_IFDIR|0755, st_size=4
0.000112 lstat("/mnt/var/www/html/drupal/sites/all/modules/Internet/date/date.module", {st_mode=S_IFREG|064
0.010816 open("/mnt/var/www/html/drupal/sites/all/modules/Internet/date/date.module", O_RDONLY) = 20
0.000835 fstat(20, {st_mode=S_IFREG|0644, st_size=44118, ...}) = 0
0.000069 lseek(20, 0, SEEK_CUR)    = 0
0.000063 stat("././sites/all/modules/Internet/date/date.module", {st_mode=S_IFREG|0644, st_size=44118, ...}) = 0
0.000159 close(20)                 = 0
...

As you can see, for every file included in the system there are around 10-15 lstat calls. this means that if we have ~200 files included we have ~2500 lstat calls per page. A simple sum on the strace results gave me ~8 seconds per page of lstat time, which was over a third of my test page time. see?

[user@host ~]# cat strace.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d |awk '{printf $2 "+"}'  | sed s/.$/\\n/ | bc
21.204960
[user@host ~]# cat strace.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d | fgrep lstat | awk '{printf $2 "+"}'  | sed s/.$/\\n/ | bc
8.361994

if you really want to understand this shell script let me know. It deserves a post by itself

Now we have 3 options: a) make lstat() faster. b) make php stop doing all those lstats(). c) make drupal include less files.

Well, (a) is not an option as our infrastructure team (aka sysadmins) insist there is nothing wrong with the filesystem and nothing can be done to enhance it's performance. Since I gave up being a sysadmin when it was still call 'sysadmin' it would be hard for me to prove them wrong (especially since they might be are right).

As (c) would require us to refactor most of our code in a very very 'not drupal way' and create very large php files, I was hoping to find a better solution.

By a rare chance of good fortune i came to learn about realpath_cache_size. Once I changed the foolish default to a reasonable 1M in php.ini, I lost most of the lstat() calls and over 10 seconds of processing time.

[user@host ~]# cat strace.2.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d |awk '{printf $2 "+"}'  | sed s/.$/\\n/ | bc
9.752309
[user@host ~]# cat strace.2.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d | fgrep lstat | awk '{printf $2 "+"}'  | sed s/.$/\\n/ | bc
.040935

Here is what to do in php.ini

; ...php.ini...
; Determines the size of the realpath cache to be used by PHP. This value should
; be increased on systems where PHP opens many files to reflect the quantity of
; the file operations performed.
 realpath_cache_size=1M

; Duration of time, in seconds for which to cache realpath information for a given
; file or directory. For systems with rarely changing files, consider increasing this
; value.
realpath_cache_ttl=300

For normal pages under normal load I was able to get a 10%-25% performance improvment. I think that's quite a bit for a configuration change.

drupal memcache items disappearing from the cache

The scenario: you are using memcached as your Drupal cache system. You do a cache_set for some item. You do a cache_get for this item. The item is there. You do a cache_get again after a few seconds/minutes, the item is not there.

You might be experiencing this issue of the memcache module: In short it means that the memcache module flushes the entire cache cluster whenever someone does a wildcard cache_clear_all. Since it's very likely that all your bins are in a single cluster you get to a situation that whenever someone-on-the-other-side-of-the-world does something that is probably completely unrelated to you, your item will still get flushed from the cache.

Note that since flush does not activly removes the items from the cache but rather marks them as expired you will not see any memcached stats that will hint to this issue.

I have commented on the issue. Here are some highlights:

...we are currently using gaolei's patch which is fast albeit expensive. We are using it on a large production system that depends heavily on memcache and we did not see any problems yet. (thank you gaolei).

We have tried to implement a lock-add-unlock scenario such as was suggested in several comments but this will be a definite nightmare for high-traffic/high-update sites...

I would suggest letting the user choose which flush mechanism to use: the current one, or the salt one. I see no reason why the module's developers should decide for me. On a site with little updates and little memory i would prefer to have the whole memcache flush; On a site with many updates and tons of memory i'm willing to sacrifice space for the sake good performance. a simple memcache_flush_method variable would do the job just fine.

Drupal SMTP Authentication with PEAR::Mail

If you want to use smtp authentication with Drupal you can use PEAR::Mail with drupal_mail_wrapper.

1. Install PEAR::Mail. On linux you can just run the following command as root:
   

    pear install --alldeps Mail

   
   


2. Create a new file, say includes/smtpmail/smtpmail.inc and paste the following code into it:
   

   
   <?php
   require_once 'Mail.php';
   require_once 'PEAR.php';
   
   function drupal_mail_wrapper($mailkey, $to, $subject, $body, $from, $headers) {
     global $smtpmail;
   
     $headers['From']    = $from;
     if ( empty($headers['To'])) {
       $headers['To'] = $to;
     }
     $headers['Subject'] = $subject;
   
     $recipients = $to;
     $mail_object =&Mail::factory('smtp', $smtpmail);
     $output = $mail_object->send($recipients, $headers, $body);
     return $output;
   }
   

   
   


3. Edit your settings.php and add the following at the end:
   

   
   $conf = array(
     'smtp_library' => 'includes/smtpmail/smtpmail.inc'
   );
   global $smtpmail;
   $smtpmail= array();
   $smtpmail["host"]  = 'your.mail.host';
   $smtpmail["auth"] = TRUE;
   $smtpmail["username"] = 'your_user_name';
   $smtpmail["password"] = 'your_password';
   

   
   

From now on every call to drupal_mail will go through your drupal_mail_wrapper and will be sent using PEAR::Mail with SMPT authentication.

I'm sure some more serious drupaler would have made this a nice module, with configuration screens and the lot, but I'm too lazy today.

phpize: command not found

If you ever get phpize: command not found error when trying to install a PEAR package on a linux system, this is probably becuase the php-devel package is not installed.

PHP Allowed memory size of 8388608 bytes exhausted

If you try to install PHP PECL package and get an "Allowed memory size of 8388608 bytes exhausted" you should know that pecl ignores php.ini memory_limit directive. As suggested in
this post, you should use pear instead of pecl to install you package. For example, instead of doing

[root@hostname tmp]# pecl -v install APC

you should do

[root@hostname tmp]# pecl -v download APC
[root@hostname tmp]# pear -v install APC-3.0.15.tgz

Updateing your docroot from subversion

We need to update our integration environment from svn every once in a while. At the moment a manual process is required since our svn gets commited into even if not everything is really stable and we only want to update the integration env when we feel the system is stable. We could do it manually by ssh-ing into the machine, but i hate people logging into my box to do stuff that should have a web interface, and we will probably want to extend that interface in the future.

Here is what I did:

First, manually do an svn checkout in the docroot:

su -
cd /var/www/html
svn co --username myusername svn://mysvnhost/svn/repositories/repository-name/projectname projectname
chown -R apache:apache projectname

Second, you will want a small PHP script to do all the work for you, so you will not have to ssh to machine all the time:

<?php
$docroot="/var/www/html";
$project="projectname";
$username="username";
$password="password";
$log = "ACTION LOG\n---------------\n\n";
$log .= shell_exec("sudo svn up --username $username --password $password $docroot/$project");
echo "<pre>$log</pre>";
?>

The thing is that since apache does not change it's user properly, svn cannot be properly run as apache, so will need the sudo there.

Third, in order for all of this to work, you will need to setup /etc/sudoers properly so go ahead and add this to it:

apache  ALL=NOPASSWD:/usr/bin/svn

This will of course grow into something much more elaborate, that allows you to select projects, update on schedule, etc.

---
EDIT
---
note that if you do what i specified above, then every update from subversion will reset the owner for the files under the docroot to root. This is generally not a good idea, so i suggest you put the svn update line in a shell script and add a "chown -R apache:apache projectname" to that script. then you can just sudo yourscript.sh from the php file.