drupal memcache items disappearing from the cache

The scenario: you are using memcached as your Drupal cache system. You do a cache_set for some item. You do a cache_get for this item. The item is there. You do a cache_get again after a few seconds/minutes, the item is not there.

You might be experiencing this issue of the memcache module: In short it means that the memcache module flushes the entire cache cluster whenever someone does a wildcard cache_clear_all. Since it's very likely that all your bins are in a single cluster you get to a situation that whenever someone-on-the-other-side-of-the-world does something that is probably completely unrelated to you, your item will still get flushed from the cache.

Note that since flush does not activly removes the items from the cache but rather marks them as expired you will not see any memcached stats that will hint to this issue.

I have commented on the issue. Here are some highlights:

...we are currently using gaolei's patch which is fast albeit expensive. We are using it on a large production system that depends heavily on memcache and we did not see any problems yet. (thank you gaolei).

We have tried to implement a lock-add-unlock scenario such as was suggested in several comments but this will be a definite nightmare for high-traffic/high-update sites...

I would suggest letting the user choose which flush mechanism to use: the current one, or the salt one. I see no reason why the module's developers should decide for me. On a site with little updates and little memory i would prefer to have the whole memcache flush; On a site with many updates and tons of memory i'm willing to sacrifice space for the sake good performance. a simple memcache_flush_method variable would do the job just fine.

MYSQL Query Optimization: Avoiding ORs

So, you've probably heard that using ORs in your queries is heavy and should be avoided when possible. Here is a live example. We had the following query running on a medium sized table in MYSQL (around 200K rows):

SELECT count(*)  FROM interactions p
   WHERE (
       (p.employer = 69 AND p.flag1 = 1)
       OR
       (p.employee = 69 AND p.flag2 = 0)
       )

Let's think of interactions as a table holding the interaction between employers and employees, and this query should tell the user the total number of interactions where user number 69 is involved.

The query looked pretty innoccent on an idle MYSQL server taking around 200ms, but when the server got busy the query execution time was reaching 5-6 seconds.

Altough we did have indexes on interactions.employer and interactions. employee doing EXPLAIN showed that MYSQL was not using them.

After some digging and fiddling and trying we ended up with:

SELECT count(*) as count FROM (
   SELECT 1  FROM interactions p
       WHERE (p.employer = 69 AND p.flag1 = 1)
   UNION ALL
   SELECT 1  FROM privatemsg p
       WHERE (p.employee = 69 AND p.flag2 = 0)
) t2

This query took on the busy server less then 250 milliseconds and less then 80ms on the idle server. Doing EXPLAIN showed that MYSQL was doing two queries, and using the correct index on each one.

After some more digging we noticed that if we EXPLAIN the original query on the idle server the optimizer occasionally converts it to a UNION, but for some reason this did not always happen and in any case took about twice the time then the UNION query.

To sum up the results here is a simple table

Query type	Server activity	Query Time
Using OR	Idle	200ms
Using OR	Busy	5500ms
Using UNION	Idle	78ms
Using UNION	Busy	189ms

Conclusion: always consider alternatives to OR, but make sure you check them well against real-time examples.

A Further Note: When timing queries in MYSQL alway use the optimizer hint /*! SQL_NO_CACHE */ to make sure you are not getting results from the query cache (if you have one set up).

Internet Explorer does not support http vary header

It was always assumed, but I just saw this quote from Microsoft:

Internet Explorer does not fully implement the VARY header per Requests for Comments (RFC) 2616. The Internet Explorer implementation of VARY is that it does not cache any data except for Vary-Useragent

Internet explorer cannot open the internet site: operation aborted

During an integration with a third party provider we were unfortunate enough to get the error

Internet explorer cannot open the internet site http://localhost: operation aborted

Yes, we have read http://support.microsoft.com/kb/927917 and did move the 3rd-party's script to be just before the </BODY> tag but to no avail.

After a long and hard effort by this provider they pointed out that while the view-source indeed looked like the script is just before </BODY> when looking at the DOM itself (they used Dominspector, I used the IE Developer Toolbar) it showed that the script was inside a <DIV> element.

As it turns out there was some code on the page that wrote an unclosed <DIV>. This made IE "fill in the blanks" and guess (wrongly) where it should place the closing </DIV> Fixing this javascript made the error go away on most pages. But not all.

I found out that the broken pages used some JQuery plugins (tooltips, jqmodal, etc.) to produce fancy decorations and effects. These scripts attached to $document.ready and did $('body').appendTo(...). This effectively added a couple of items to the DOM between the 3rd-party script and the </BODY>. I am still not quite sure why this should cause IE to choke but since we have a tight schedule we simply changed those scripts to add their stuff to some other elements. This indeed solved out problem.

They funny (or maybe sad) thing was that on Firefox we did not get any error but the page simply disappeared. Chrome worked just fine.

Misplaced elements with position:relative

We had a situation where we had three <DIV> elements in a column and the middle element had content dynamically loaded into it after the page has completed loading. When the middle element finished loading, some of the content of the last element seemed to forget its parenthood and floated nicely over the second element.




Naturally this only happened in Internet Explorer (what else).

After a lot of digging we noticed that the misplaced content of <DIV>#3 had a position:relative style (indicated as #4 in the image above). It is "well known" that IE does not handle relative positioning properly when the layout of the page change.

Luckily I chanced upon an article by Holly Bergevin et al. called On having layout which was enlightening. Understanding that IE will only respect the element's state if it have the hasLayout property quickly solved the problem.

In our specific case I used display:inline-block to force <DIV>#3 to have hasLayout set to true, which made IE respect the element's style and re-render the element's content (i.e. <DIV>#4) after re-positioning it. My only concern now is what effect this might have on rendering performance, but it will have to do for now.

Tip: you can use IE Developer Toolbar to check if an element hasLayout. It will show the hasLayout property as set to -1 if hasLayout is true.

Further credit is due to Ingo Chao who wrote relatively positioned parent and floated child – disappearance.

Absolute vs. Relative URLs and SEO

I've seen several places where people suggest that one should use absolute URLs (http://domain.com/page) to do internal linking instead of relative URLs (/page). Those people relate to a google article where it is quoted that

We also suggest you link to other pages of your site using absolute, rather than relative, links with the version of the domain you want to be indexed under. For instance, from your home page, rather than link to products.html, link to http://www.example.com/products.html . And whenever possible, make sure that other sites are linking to you using the version of the domain name that you prefer

This seems to me a classic example of taking things out of context. If you read the entire article you can see that Ms. Fox is talking about a specific case where your site can be accessed by more then one domain name (e.g. http://www.domain.com and http://domain.com).

There is nothing google ever wrote that i could find that say that absolute URLs are better if your site is only accessed by one domain name.

There is one exception i can think of: if your domain is "coolstuff.com" for example and you do use absolute URLs, then the word "coolstuff" will appear in your pages alot. This might be something that may boost your ranking with regards the the word "coolstuff". But this is just a guess.

More reading: Google Canonicalization Problems? - Crawling, indexing, and ranking | Google Groups

Please note: this is my personal opinion. I've had at least two SEO experts that claim that absolute URLs will give better performance on google ranking. Since I'm strong-headed I will hold to my opinion until I'll be proven (with numbers) otherwise. Feel free to comment with supporting or conflicting opinions and data.

Save binary file in Tcl under V6

This issue pops up every couple of years. It's nothing new, but still worth documenting.

The Challenge: To save a posted file to a file system, not using SUBMIT_STATIC_FILE.

The Problem: Vignette V6 pre-process the form submitted data so that any <input type="file"> are encoded as an hex string. If you ERROR_TRACE the variable for the file, you will see a string that looks like 0x1D00ABADD1D9....

The Solution: binary format to the rescue. and with a bit of trimming, you get it all in a few lines:

proc save_files { } {
#FileData,FileExtention,Field1 are posted from the <form>

set filename [generate_filename]
set bin_filename "${filename}.[SHOW FileExtention]"
set xml_filename "${filename}.xml"

### This will save a binary file.
set file [open $bin_filename "w"]
fconfigure $file -translation binary
puts -nonewline $file [binary format H* [string range [SHOW FileData] 2 end-1]]
close $file

### This will save a utf-8 xml (text) file
set file [open $xml_filename "w"]
fconfigure $file -encoding "utf-8"
puts $file "<?xml version='1.0' encoding='utf-8' ?>"
puts $file "<formdata>"
puts $file "<Field1>[HTML_ESCAPE [SHOW Field1]]</Field1>"
#...more fields...
$puts $file "<Attachment>$bin_filename</Attachment>"
puts $file "</formdata>"
close $file
}

* Remember to use H* and not h*.
* The example above also saves a text file in utf-8 with extra data from the form
* Make sure your form is enctype=multipart/form-data
* This will not work in Storyserver 4.2.

Make a Footer Stick to the Bottom of the Page

Ryan Fait wrote a great piece of code that makes a footer stick to the bottom of the page with css only. no javascript, no hassle. genius, absolutly genius.

Apache Rewrites and SetEnv

It seems that variables set with Apache's SetEnv (and hence SetEnvIf and BrowserMatch) are being ignored by RewriteRule and RewriteCond. The rumor has it to be a design issue. The only way i could find around this problem is to use the [E= rewrite rule directive.

#Use RewriteRule & RewriteCond as a replacment for SetEnv and BrowserMatch
#SetEnv browser="-ie"
RewriteRule ^(.*)$ - [E=browser:-ie]
#BrowserMatch "IE7" browser="-ie7"
RewriteCond %{HTTP_USER_AGENT} "MSIE 7"
RewriteRule ^(.*)$ - [E=browser:-ie7]
#BrowserMatch "Firefox" browser="-moz"
RewriteCond %{HTTP_USER_AGENT} "Firefox"
RewriteRule ^(.*)$ - [E=browser:-moz]

RewriteCond %{DOCUMENT_ROOT}/cache/index%{ENV:browser}.html -f
RewriteRule ^(.*)$ cache/index%{ENV:browser}.html [L]

PS. If you need to debug rewrite rules you can add the following lines to your httpd.conf:
RewriteLog logs/rewrite.log
RewriteLogLevel 5

Annoying FireBug Bug

I came across an annoying bug in firebug. If there is a global variable with the same name as a local variable then firebug does not correctly show the value of the local variable in the tooltips and the watch window.

The most annoying thing is that it has been reported several months ago, but still no fix.

UPDATE: Confirmed Fixed in 1.2 on FF3.

Drupal SMTP Authentication with PEAR::Mail

If you want to use smtp authentication with Drupal you can use PEAR::Mail with drupal_mail_wrapper.

1. Install PEAR::Mail. On linux you can just run the following command as root:
   

    pear install --alldeps Mail

   
   


2. Create a new file, say includes/smtpmail/smtpmail.inc and paste the following code into it:
   

   
   <?php
   require_once 'Mail.php';
   require_once 'PEAR.php';
   
   function drupal_mail_wrapper($mailkey, $to, $subject, $body, $from, $headers) {
     global $smtpmail;
   
     $headers['From']    = $from;
     if ( empty($headers['To'])) {
       $headers['To'] = $to;
     }
     $headers['Subject'] = $subject;
   
     $recipients = $to;
     $mail_object =&Mail::factory('smtp', $smtpmail);
     $output = $mail_object->send($recipients, $headers, $body);
     return $output;
   }
   

   
   


3. Edit your settings.php and add the following at the end:
   

   
   $conf = array(
     'smtp_library' => 'includes/smtpmail/smtpmail.inc'
   );
   global $smtpmail;
   $smtpmail= array();
   $smtpmail["host"]  = 'your.mail.host';
   $smtpmail["auth"] = TRUE;
   $smtpmail["username"] = 'your_user_name';
   $smtpmail["password"] = 'your_password';
   

   
   

From now on every call to drupal_mail will go through your drupal_mail_wrapper and will be sent using PEAR::Mail with SMPT authentication.

I'm sure some more serious drupaler would have made this a nice module, with configuration screens and the lot, but I'm too lazy today.

"The system cannot execute the specified program" Error

You are trying to run some program on Windows (such as apache.exe or htpasswd.exe) and you are getting "The system cannot execute the specified program" error. This usually means that the program you are trying to run was compiled against DLLs that are not on your system. You should use Dependency Walker to see which DLLs are missing, and then install them.

The Apache 2.x binary windows distribution, specifically, was compiled against the Visual Studio 2005 re-distributable package, which you can download from microsoft.

Further reading (1) and (2)

Edit (Nov 2011):
Following this comment (by Annonymous) please note that you may need to download and use the 2008 package and not the 2005 one: The official download for the Microsoft Visual C++ 2008 SP1 Redistributable Package (x86) or The official download for the Microsoft Visual C++ 2008 Redistributable Package (x64)

Remote Root Access to MySql

To help me remember:

mysql -u root
mysql> SET PASSWORD FOR 'ROOT'@'LOCALHOST" = PASSWORD('new_password');
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'password' WITH GRANT OPTION;
mysql> FLUSH PRIVILEGES;
mysql> exit;

Source: ben robison :: Howto: Remote Root Access to MySql

phpize: command not found

If you ever get phpize: command not found error when trying to install a PEAR package on a linux system, this is probably becuase the php-devel package is not installed.

Uninstall Google Desktop Gadget

In order to uninstall any Google Desktop gadget you can do the following:
1. Close Google Desktop
2. Open 'My Documents' folder and locate and open 'My Google Gadgets' folder in it
3. Find the gadget you want to remove and delete it.
4. Open Google Desktop again.

(source: uninstall RSStoSpeech Google Desktop gadget - RSS To Speech Gadget | Google Groups)

document.body.scrollTop in IE

It seems that document.body.scrollTop does not work in IE6 if the document's doctype is defined as

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

To find out the scrolling offset in a way that works for both DTD3 and DTD4.01 you can use

(document.documentElement.scrollTop?document.documentElement.scrollTop:document.body.scrollTop)

Same goes for scrollLeft.
See more at document.body.scrollTop in IE

Blog Archive Carousel Widget

I've played around with blogspots' widget system and came up with a nice blog archive carousel widget. The widget is based on the standard blogger.com blog archive widget and jcarousel, a jquery plugin.
DISCLAIMER: Use at your own discretion.
How to use:

• Go to your blog's Template -> Edit HTML.


• Download Full Template for backup and save it somewhere you'll remember.


• Go go Template -> Page elements and add a blog archive widget, save the page.


• Go back to Template -> Edit HTML.


• Make sure Expand Widget Templates is not checked.


• Just below the <data:blog.pagetitle/> add the code below
  

  
   <!-- archive carousel -->
   <script src='http://javajavaproxy.googlepages.com/jquery-1.2.1.min.js'/>
   <script src='http://javajavaproxy.googlepages.com/jquery.jcarousel.pack.js'/>
   <script type='text/javascript'>
   jQuery(document).ready(function() {
       jQuery('#mycarousel').jcarousel({  vertical: true,  scroll: 3}); });
   </script>
   <!-- //archive carousel -->
  



• At the end of the <skin> section, just above the ]]></b:skin>add the code below
  

  
  /* new archive */
  UL.posts LI {  font-size:80%;  width:170px;  padding-right:2px;  border-bottom:1px solid silver;}
  .jcarousel-container { position: relative; }
  .jcarousel-clip { z-index: 2; padding: 0; margin: 0; overflow: hidden; position: relative; }
  .jcarousel-list { z-index: 1; overflow: hidden; position: relative; top: 0; left: 0; margin: 0; padding: 0; }
  .jcarousel-item { float: left; list-style: none; width: 170px; xheight: 18px; }
  .jcarousel-next { z-index: 3; display: none; }
  .jcarousel-prev { z-index: 3; display: none; }
  .jcarousel-skin-tango.jcarousel-container { -moz-border-radius: 10px; background: #F0F6F9; border: 1px solid #346F97; }
  .jcarousel-skin-tango.jcarousel-container-vertical { width: 170px; height: 170px; padding: 40px 20px; }
  .jcarousel-skin-tango .jcarousel-clip-vertical { width:  190px; height: 170px; }
  .jcarousel-skin-tango .jcarousel-item { width: 170px; xheight: 18px; }
  .jcarousel-skin-tango .jcarousel-item-vertical { margin-bottom: 10px; }
  .jcarousel-skin-tango .jcarousel-item-placeholder { background: #fff; color: #000; }
  .jcarousel-skin-tango .jcarousel-next-vertical { position: absolute; bottom: 5px; left: 99px; width: 32px; height: 32px; cursor: pointer; background: transparent url(http://javajavaproxy.googlepages.com/next-vertical.png) no-repeat 0 0; }
  .jcarousel-skin-tango .jcarousel-next-vertical:hover { background-position: 0 -32px; }
  .jcarousel-skin-tango .jcarousel-next-vertical:active { background-position: 0 -64px; }
  .jcarousel-skin-tango .jcarousel-next-disabled-vertical,
  .jcarousel-skin-tango .jcarousel-next-disabled-vertical:hover,
  .jcarousel-skin-tango .jcarousel-next-disabled-vertical:active { cursor: default; background-position: 0 -96px; }
  .jcarousel-skin-tango .jcarousel-prev-vertical { position: absolute; top: 5px; left: 99px; width: 32px; height: 32px; cursor: pointer; background: transparent url(http://javajavaproxy.googlepages.com/prev-vertical.png) no-repeat 0 0; }
  .jcarousel-skin-tango .jcarousel-prev-vertical:hover { background-position: 0 -32px; }
  .jcarousel-skin-tango .jcarousel-prev-vertical:active { background-position: 0 -64px; }
  .jcarousel-skin-tango .jcarousel-prev-disabled-vertical,
  .jcarousel-skin-tango .jcarousel-prev-disabled-vertical:hover,
  .jcarousel-skin-tango .jcarousel-prev-disabled-vertical:active { cursor: default; background-position: 0 -96px; }
  



• find the
  <b:widget id="'BlogArchive1'" locked="'false'" title="'Blog" type="'BlogArchive'/">
  and replace this line with the following code
  

  
  <b:widget id='BlogArchiveCarousel' locked='false' title='Blog Archive' type='BlogArchive'>
  <b:includable id='toggle' var='interval'>
    <b:if cond='data:interval.toggleId'>
    <b:if cond='data:interval.expclass == "expanded"'>
      <a class='toggle' expr:href='data:widget.actionUrl + "&amp;action=toggle" +       "&amp;dir=close&amp;toggle=" + data:interval.toggleId +       "&amp;toggleopen=" + data:toggleopen'>
          <span class='zippy toggle-open'>&#9660; </span>
      </a>
    <b:else/>
      <a class='toggle' expr:href='data:widget.actionUrl + "&amp;action=toggle" +         "&amp;dir=open&amp;toggle=" + data:interval.toggleId +         "&amp;toggleopen=" + data:toggleopen'>
            <span class='zippy'>
              <b:if cond='data:blog.languageDirection == "rtl"'>
                &#9668;
              <b:else/>
                &#9658;
              </b:if>
            </span>
      </a>
    </b:if>
   </b:if>
  </b:includable>
  <b:includable id='interval' var='intervalData'>
      <b:loop values='data:intervalData' var='i'>
       <b:if cond='data:i.data'>
         <b:include data='i.data' name='interval'/>
       </b:if>
       <b:if cond='data:i.posts'>
         <b:include data='i.posts' name='posts'/>
       </b:if>
      </b:loop>
   </b:includable>
  <b:includable id='menu' var='data'>
    <select expr:id='data:widget.instanceId + "_ArchiveMenu"'>
      <option value=''><data:title/></option>
      <b:loop values='data:data' var='i'>
        <option expr:value='data:i.url'><data:i.name/> (<data:i.post-count/>)</option>
      </b:loop>
    </select>
  </b:includable>
  <b:includable id='flat' var='data'>
    <ul>
      <b:loop values='data:data' var='i'>
        <li class='archivedate'>
          <a expr:href='data:i.url'><data:i.name/></a> (<data:i.post-count/>)
        </li>
      </b:loop>
    </ul>
  </b:includable>
  <b:includable id='posts' var='posts'>
       <b:loop values='data:posts' var='i'>
         <li>&amp;nbsp;<a expr:href='data:i.url'><data:i.title/></a></li>
       </b:loop>
   </b:includable>
  <b:includable id='main'>
      <b:if cond='data:title'>
        <h2><data:title/></h2>
      </b:if>
      <div class='widget-content'>
      <div id='ArchiveList'>
       <div expr:id='data:widget.instanceId + "_ArchiveList"'>
         <ul class='jcarousel-skin-tango posts' id='mycarousel'>
            <b:include data='data' name='interval'/>
         </ul>
       </div>
      </div>
      <b:include name='quickedit'/>
      </div>
   </b:includable>
  </b:widget>
  



• Preview the template


• If all looks well, you can save your template. You might get prompted to confirm that BlogArchive1 will be deleted. Allow it (we changed it's name to "BlogArchiveCarousel")

MySQL 5.1 Clustering Might Just Actually Work

As I previously wrote mysql 5.0 clustering was a big issue for us since all data as placed in memory. It seems that mysql 5.1 clustering is going to be much better, including the ability to store non-index columns on disk (indexes and indexed columns are still stored in-memory), do (limited) cluster-to-cluster replication, and better varchar support for memory columns. Pitty it's only release candidate at the moment. can't wait for the GA to come out.

Tomcat java.net.BindException: Cannot assign requested address

I had a very strange error today when I tried to startup tomcat on some linux box that used to work. Since google was no help I figured I should document this somewhere.

Tomcat startup gave the notorious java.net.BindException: Cannot assign requested address. This usually means that another process is listening on the same port on this machine. I started with ps -ef grep java to make sure that no other tomcat process was running on the machine. Nothing.

I suspected that maybe some other web server was holding the port, but netstat -an showed that nothing binds this address. Moreover, telnet localhost 80 showed that no one is listening.

Looking at the server.xml I found out that the connector was binding to the hostname "demoserver" on port 80.

<Connector className="org.apache.coyote.tomcat4.CoyoteConnector"
address="demoserver" port="80" 
minProcessors="5" maxProcessors="75"
enableLookups="true" redirectPort="8443"
acceptCount="100" debug="0" connectionTimeout="20000"
useURIValidationHack="false" disableUploadTimeout="true" />

trying to ping demoserver gave me errors, but nslookup demoserver gave me the right ip address.

After a few minutes I noticed that some clown added demoserver to /etc/hosts with the wrong ip address. Fixing /etc/hosts to have the right ip address for demoserver made tomcat work smooth and nice.

PHP Allowed memory size of 8388608 bytes exhausted

If you try to install PHP PECL package and get an "Allowed memory size of 8388608 bytes exhausted" you should know that pecl ignores php.ini memory_limit directive. As suggested in
this post, you should use pear instead of pecl to install you package. For example, instead of doing

[root@hostname tmp]# pecl -v install APC

you should do

[root@hostname tmp]# pecl -v download APC
[root@hostname tmp]# pear -v install APC-3.0.15.tgz

Java TTL Cache, the Quick and Dirty Way

There are many caching systems in java. I really like some of them (especially Ian Schumacher's LRU Cache). But sometimes you just to drop a quick few lines to get something in a local cache. Here is one way of doing it:

private static Map urlItemCache = Collections.synchronizedMap(new HashMap());
private static Map urlItemCacheTtl = Collections.synchronizedMap(new HashMap());
private static final long TTL=1000*60*60;//one hour in milliseconds

public static String findItemByUrl( String url) {
 String item="";
 Long now=Long.valueOf(Calendar.getInstance().getTimeInMillis());

 //try to get the item from the local cache
 item = urlItemCache.get(url);
 if ( !isEmpty(item)) {
  //urlItemCacheTtl.get(url) is guaranteed to have a value, since it's added at the same time urlItemCache is.
  if (now.longValue() - urlItemCacheTtl.get(url).longValue() < TTL ) {
   urlItemCacheTtl.put(url, now); //see comment below
   return item;
  } else {
   urlItemCache.remove(url);
   urlItemCacheTtl.remove(url);
  }
 }
 //not found in cache
 //... do you application magic here to get the item ...
 if ( !isEmpty(item) ) {
  urlItemCache.put(url, item); //add to local cache
  urlItemCacheTtl.put(url, now);
  return item;
 } //else 
 return null;
}

Reload webapp from ant script

A co-worker asked me how he can reload a web-application from an ant script. GDS was very helpfull finding the line in an old ant script I used to use. In the example below you might want to define $project.output

<property name="project.output" value="WEB-INF/classes"/>
<!-- reload the current webapp to refresh the classes that we just changed -->
<get 
 src="http://localhost:8080/manager/reload?path=/your-web-app" 
 username="admin-user" 
 password="admin-pass"
 dest="${project.output}/log/reload.log" 
/>
<concat><fileset dir="${project.output}/log" includes="reload.log"/></concat>    

RequestDispatcher.forward() and filters

If you have read this post then you probably know that if you use RequestDispatcher.forward() (or RequestDispatcher.include()) the filter chain does not get re-called for the forwarded servlet/jsp. However, if you read more closely you see that Craig was talking about the 2.3 servlet spec. Since we are using tomcat 5.5 which conforms to the 2.4 spec, you should know that there's a new <dispatcher> element in the deployment descriptor with possible values REQUEST, FORWARD, INCLUDE, and ERROR. You can add any number of entries to a <filter-mapping> tag and individually decide which filter gets called when. An Example:

    <filter-mapping>
        <filter-name>UrlRewriteFilter</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>

    <filter-mapping>
        <filter-name>SecurityFilter</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>FORWARD</dispatcher>
    </filter-mapping>

Simpler GUI for Jad, The Java Decompiler from As I Learn

This is definitely the most useful two-liner I've seen in a while:
Simpler GUI for Jad, The Java Decompiler.

I naturally customized it to:

jad -o -s java -lnc -d%TEMP% %1
"C:\Program Files\TextPad 5\TextPad.exe" %TEMP%.\%~n1.java

Changed extension to java so that textpad does syntax highlighting; added -lnc so that jad outputs the original line numbers as comments and I can trace the exceptions I get

(after a few questions from people i'm editing this post:)
You will need to download jad and extract the exeutable to somewhere on your path, say C:\windows (or c:\winnt). Then create a new .cmd file, paste the above into the .cmd file, and put it somewhere. Then rightclick any .class file you can find, choose "Open With..."->"Choose Program"->"Browse..." and find the cmd file you created (don't forget to check the "Always use this program"). This way, any .class you'll double-click will decompile and open in textpad.

Using java regular expressions to match groups

It's been demonstrated over and over again, but I always seem to forget the "simple example". So here it is:

import java.util.regex.Matcher;
import java.util.regex.Pattern;
...
/*
 *Going to match a URL that looks like this 
 * /url/prefix/--sitename--/menu.--menuid--/?param=--itemid--
 * and change it into 
 * /url/prefix/--sitename--/--menuname--/--itemname--
*/
String expr="(/url/prefix/[a-zA-Z0-9_-]+)/menu\\.([a-zA-Z0-9_-]+)/\\?param=([a-zA-Z0-9_-]+)";
Pattern p = Pattern.compile(expr);
Matcher m = p.matcher(uri);
if ( m.matches() ) {
    prefix= m.group(1);
    menuitem = m.group(2);
    guid = m.group(3);
    if ( guid != null && prefix != null && menuitem != null ) {
        uribuilder.append(prefix);
        uribuilder.append("/");
        uribuilder.append(getMenuName(menuitem));
        uribuilder.append("/");
        uribuilder.append(getItemName(guid));
        uri=uribuilder.toString();
    }
}

Does GDS slows my hard disk?

I've a encountred a strange problems that when Google Desktop Search is running on my PC (which it almost always is), my hard disk's read speed is about 1/6th of the normal speed. I'll contact google to see if this is a known issue. I'm using several HD benchmark tools to test the speed, most notably Simpli Software's Hd Tach.

Here are screenshots of my disk's speed with and without GDS running.

Avoding browser history when changing iframe src

We ran into a problem where we needed to have an iframe who's source needed to be changed dynamically by a javascript based on some user data. The issue was that Internet Explorer saved the iframe's old url in the history. The simple solution is to dynamically write the iframe with it's real src instead of having an iframe on the page and then changing it's src.

You do it like this:

var iframeHeaderCell = document.getElementById('wheretoputheiframe');
var dynamicURL = 'http://...' //your url
var iframeHeader = document.createElement('IFRAME');
iframeHeader.id = 'iframeHeader';
iframeHeader.src = dynamicURL ;
iframeHeader.width = ...;
iframeHeader.height = ...;
iframeHeader.scrolling = 'no';
iframeHeader.frameBorder = 0;
iframeHeader.align = 'center';
iframeHeader.valign='top'
iframeHeader.marginwidth = 0;
iframeHeader.marginheight = 0;
iframeHeader.hspace = 0;
iframeHeader.vspace = 0;
iframeHeaderCell.appendChild(iframeHeader);

Elementry, my dear Watson.

Updateing your docroot from subversion

We need to update our integration environment from svn every once in a while. At the moment a manual process is required since our svn gets commited into even if not everything is really stable and we only want to update the integration env when we feel the system is stable. We could do it manually by ssh-ing into the machine, but i hate people logging into my box to do stuff that should have a web interface, and we will probably want to extend that interface in the future.

Here is what I did:

First, manually do an svn checkout in the docroot:

su -
cd /var/www/html
svn co --username myusername svn://mysvnhost/svn/repositories/repository-name/projectname projectname
chown -R apache:apache projectname

Second, you will want a small PHP script to do all the work for you, so you will not have to ssh to machine all the time:

<?php
$docroot="/var/www/html";
$project="projectname";
$username="username";
$password="password";
$log = "ACTION LOG\n---------------\n\n";
$log .= shell_exec("sudo svn up --username $username --password $password $docroot/$project");
echo "<pre>$log</pre>";
?>

The thing is that since apache does not change it's user properly, svn cannot be properly run as apache, so will need the sudo there.

Third, in order for all of this to work, you will need to setup /etc/sudoers properly so go ahead and add this to it:

apache  ALL=NOPASSWD:/usr/bin/svn

This will of course grow into something much more elaborate, that allows you to select projects, update on schedule, etc.

---
EDIT
---
note that if you do what i specified above, then every update from subversion will reset the owner for the files under the docroot to root. This is generally not a good idea, so i suggest you put the svn update line in a shell script and add a "chown -R apache:apache projectname" to that script. then you can just sudo yourscript.sh from the php file.

mysqltop.sh

If you need to monitor mysql proccesslist, dump the following to a shell script and chmod 755. (ofcourse you will need to add passwords etc. if you have any;)

#!/bin/sh

watch 'mysql<<EOF
show processlist;
EOF
'

(Control+C will let you out)

Two webapps using shared jar - ClassLoader problem

Yeah, yeah, I know it's obvious, but i thought i might be smarter. I am not:

Charles R Caldarale Writes:

“[When using Tomcat,] classes from different webapps cannot access each other - no ifs, ands, or buts. Anything that is to be accessed from more than one webapp must be put in a common location [such as tomcat/shared].”

See:

• http://www.mail-archive.com/users@tomcat.apache.org/msg30896.html
• http://tomcat.apache.org/tomcat-6.0-doc/class-loader-howto.html
• http://tomcat.apache.org/tomcat-5.5-doc/class-loader-howto.html

Internet Explorer CSS Limit

We just came across a bug that makes IE choke when it needs to handle over 30 <style> objects. If you thought of using a single style object and then @import the other files, it won't work either.

You might want to ask why we need over 30 style objects. Well, since each drupal module tends to add it's own CSS (using drupal_add_css) we very quickly got to 22 CSS's being included. From there the road to 30-32 was quite trivial. Guess we'll have to use Drupal's CSS/JS aggragator to solve this.

Calcualting Average in UNIX Shell Script

I grabbed a piece of code to calculate an average for data in shell. It didn't work so I had to modify it. This is the result.

#!/bin/sh
n=0
sum=0
while read x
do
        sum=`echo $sum + $x | bc`
        if [ "$?" -eq "0" ]; then
                n=`expr $n + 1`
        fi
done
echo "scale=2;$sum/$n"| bc

Server Header in Apache Tomcat

Altough you cannot disable the header, you can change it to anything you like by adding a "server" attribute to the connector tag in server.xml
e.g.

    <connector 
    port="80" address="localhost" 
    ...
    server="Undefined"
    >

RoundToNearest in Microsoft Excel

This will round the given cells to the nearest number given, pretty neat if you want to produce a repost that is rounded, e.g. to the 0.25.

=IF(C50>0,CEILING(C50,0.25),-CEILING(C50,0.25))

MySQL Cluster

I did some research on MySQL cluster so i can recommend one of our customers whether to use it or not. It turns out that the MySQL cluster is very different from other DBMS clustering solutions in that it distributes the data across the nodes (synchronously) and then stores it on the disk (asynchronously). This is all very nice, but at the moment all data is stored in memory, so if I have a 120GB database I would need around 150GB memory on each node, which is silly.

The Wikipedia Page on MySQL Cluster gives more details.

Waiting for MySQL to get better...

UPDATE: It seems that this is no longer the case for the 5.1 version of mysql. See here.

Welcome - eureka

Why this blog?

Post random thoughts about random technology issues. Basically acts as an online(public) notepad, so i can keep track of things i found and do not wish to forget.

Why eureka?

eureka is the greek word for 'I have found (it)', supposedly exclaimed
by Archimedes upon discovering how to measure the volume of an irregular solid
and thereby determine the purity of a gold object. source: http://www.answers.com/eureka&r=67