Wednesday, January 20, 2016

Timing django/python function calls

Doing some optimization for I've come across a handy snippet to time the execution of python/django functions. Since I was doing deep-dive checking I also wanted to know the stack of the executed function. Figured someone (me) would want to use it in the future, so I'm adding it here. Not the prettiest of code, but it works.

import time
import inspect                                                
def timeme(method):
    def time_wrapper(*args, **kw):
        startTime = int(round(time.time() * 1000))
        result = method(*args, **kw)
        endTime = int(round(time.time() * 1000))
        for i in inspect.stack():
            if i[3] == 'time_wrapper': continue 
        print 'Time(ms): ', ('%04d')%(endTime - startTime), method.__name__, u'->' ,u'->'.join(reversed(stack[0:5]))
        #only show 5 top stack
        return result
    return time_wrapper

#usage example
def some_function(x,y):
    return something

Tuesday, July 30, 2013

Setting I18N Hebrew in Django 1.5

It should be simple. But the docs are not very clear. So here is how to setup your Django site to be in Hebrew (or any other language).

The need: Make my site work in Hebrew, without taking into account Browser's Accept Language and all this stuff.

Ingredients: Working Django 1.5 project on Linux, common sense


Install GNU gettext (even if you think you have it already)

sudo apt-get install gettext
Change in projectname/

# Language code for this installation. All choices can be found here:
# LANGUAGE_CODE = 'en-us'
# This will force this installation to be in Hebrew


# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True

# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale.
USE_L10N = True

# If you set this to False, Django will not use timezone-aware datetimes.
USE_TZ = True
Make sure your MIDDLEWARE_CLASSES do not have LocaleMiddleware. If you do have LocaleMiddleware than Django will try to guess the right locale from the request context, which we don't want at the moment.

    #Comment out LocaleMiddleware if it's there to keep in the same locale all the time
    # Uncomment the next line for simple clickjacking protection:
    # 'django.middleware.clickjacking.XFrameOptionsMiddleware',
You may also need to add django.core.context_processors.i18n to your TEMPLATE_CONTEXT_PROCESSORS. (Don't do that right now, wait and see if everything is working and only add this if stuff is not working)

#add this at the end of if I18N is not working
from django.conf import global_settings
In your python code wrap translatable texts with ugettext_lazy or ugettext (you can and should alias them to some conventional shortcut (I use _t for ugettext and _ for ugettext_lazy, for example

from django.utils.translation import ugettext_lazy  as _
from django.utils.translation import ugettext  as _t
class Shirt(models.Model):
                    ('B', _('Blue')),
                    ('G', _('Green')),
    color = models.CharField(max_length=1, choices=COLOR_TYPES)
        return _t('Shirt')

In your templates you need to use this (ugly and hard to read) method

{% load i18n %}
<!-- Make sure this LANGUAGE_CODE is correct: {{LANGUAGE_CODE}} -->
<h1>{%trans "Shirt Store"%}</h1>
or you can use {{_("Shirt Store")}} as an alternative method
Remember that you need to {% load i18n %} in every template that uses {%trans%}.

Now we need to create the tranlsation files themselves:

cd django-project
cd django-app
mkdir locale makemessages -l he
vi locale/he/LC_MESSAGES/django.po compilemessages
Rinse, repeat

Run your site and hope for the best.


Thursday, August 23, 2012

"Type 'Newtonsoft.Json.Linq.JToken' is a recursive collection data contract" While Adding Service Reference in VS2012

Trying to add a service reference in Visual Studio 2012 to Bing Maps SOAP service I got the following error message:

Cannot import wsdl:portType
Detail: An exception was thrown while running a WSDL import extension: System.ServiceModel.Description.DataContractSerializerMessageContractImporter
Error: Type 'Newtonsoft.Json.Linq.JToken' is a recursive collection data contract

Following Getting “Recursive collection data contract” when referencing a WCF service with a slightly complex method and Visual Studio 2012 Error Reference.svcmap throwing Error I've had to change the "reuse types in referenced assemblies" from "all" to everything except Newtonsoft.JSON and RestSharp.

Might not be something to write home about, but I'm sure I'll forget where it is was in a year or so.

PS. This seems to be VS2012 only issue, VS2010 worked just fine.

Thursday, November 18, 2010

Tech-Ed Eilat 2010

So happy to be presenting at Tech-Ed Eilat 2010.
See you all at the Herods Hotel, Tue 30/11 at 11:15 where we would share our lessons from the field on enterprise web content management with SharePoint 2010.

You can read more about our session at the realcommerce web site.

Tuesday, January 26, 2010

IIS JNLP 404 Problem

If your IIS 6 returns a 404 Not Found when you request a Java Web Start jnlp file, then you probably need to add the jnlp mime-type to the IIS configuration.

IIS Settings -> Right click the machine -> mime-types -> new Extension: .jnlp MIME Type: application/x-java-jnlp-file

Sunday, October 4, 2009

How to do rewrites in an IIS ISAPI

Note to self: If ever you need to do a rewrite in an IIS ISAPI Filter you just need to re-set the pHeaderInfo's url header. Something like this:

strNewUrl = GET_REWRITE_FOR_URL(strUrl);
if ( strNewUrl.IsEmpty() == FALSE ) {
"url", (LPTSTR)(LPCTSTR)strNewUrl);

This is working in IIS4 and above.

Note that if you really need a good re-write/redirect filter for IIS you should look at IIRF ,but since it's not working for IIS4 i had to do this manually

Thursday, June 18, 2009

Bing, Google, Nutch and Canonical URLs

Google announced a few months back that they started supporting canonical urls. This is a great feature that we already adopted in a couple of sites (mostly to keep tracking codes from messing with our search engine results.

Unfortunelty, altough Microsoft announced they will support this feature in Live, this was not yet implemented in Bing as of now (Jun 2009).

Nutch, on the other hand, had this on thier TODO list for 1.1.

Also, Google Search Appliance (GSA), is currently assumed by all to not support this (although nobody really knows if it does).

More about canonical URLs:

Wednesday, June 3, 2009

apc futex_wait lockdown make your apache freeze over

We had the typical LAMP setup going on, with Drupal as the base CMS and APC for bytecode cache. We needed a good caching engine so I figured why not use APC's user cache. Well, we tried the APC Cache Drupal Module which, with minor fixes proved to work very nicely. That is, until we actually put this all thing on production.

The first thing we had was having our apache hang and not respond to any user requests. We susspected network issues, especially since netstat -na showed that all the apache processes were hanging on SYN_WAIT. However, since apache restart solved the issue i started to suspect this was something else.

To make a long (very long) story short, I got strace on our prod machines to find out that apache was either hanging on futex_lock(....FUTEXT_WAIT...) or doing infinte loops on the same functions.

To make even a longer story short, I got gdb installed on those machines and the backtrace clearly indicated that the locks were from APC user-cache calls.

We decided to abandon APC user-cache and switch to memcached which proved faster and had less lockdowns.

The funny thing is that when we talked about this over dinner the same evening a developer from another team just pointed me to this article by one of the APC leaders: (or something) How to Dismantle an APC Bomb which has been around for over a year. I am supprised and shocked that such a information is hidded so well and not mentioned anywere in the docs. Moreover, I went through the APC code again after reading this post (I went through it once when i started analysing the problem) and it seems that this is not even close to being resolved. there are no patches and no TODOs and nothing of the sort. From reading the code the entire user-cache needs a major re-write. What gives?

(this is a post i wrote a couple of months ago, never had time to finish it. Unfortunately this is still not fixed afaik)
EDIT (07/2009): reports this to be fixed. If anyone can confirm this please send me a note so that I could update this post

Thursday, May 14, 2009

GA_googleAddAttr is not defined error

If you are using Google Ad Manager you can define custom attributes for targeting.

If you are getting a javascript error GA_googleAddAttr is not defined just make sure to put the call(s) to GA_googleAddAttr in a seperate <script> line, after the GS_googleEnableAllServices.

<script type="text/javascript">
<script type="text/javascript">
GA_googleAddAttr("ATTRB1", "whatever");
<script type="text/javascript">
GA_googleAddSlot("ca-pub-0000000000", "slotname_216x311");

Monday, March 23, 2009

V6 DECODE64 not working properly

It turns out that after all these years, V6's built-in DECODE64 and ENCODE64 actually do not work properly with some binary data.

proc COMPARE_BASE64 {} {
set code {2lIU1ZNw5BYvKi79j4L/+GGmjAQK7uiBQG7elDdKZcE=}
set vgn_result [DECODE64 $code]; #VGN built-in function
set tcl_result [::base64::decode $code]; #TclLib tcl-only implementation
return [join [list \\
[BIN2HEX $vgn_result] \\
[BIN2HEX $tcl_result] \\
[string compare $vgn_result $tcl_result ] \\
] "\\r\\n"]
proc BIN2HEX { text } { binary scan $text H* result; return $result }

--- result ----

If you need to encode/decode base64 for use in non-vignette systems make sure you use tcllib's base64.

This was tested on Vignette's V6 but I am sure it's true for Storyserver 4.2 and V5 as well.

Thursday, March 19, 2009

MySQL indexes behaving badly

If you have a MySQL query that for no apparent reason stops working or no longer behaves as it used to, the first thing to do is EXPLAIN it. If the explain is unreasonable (for example, using different indexes then it used to or using an index but still scanning millions or rows) then you should try to CHECK TABLE.

One thing that I accidently found out was that CHECK TABLE also updates the key statistics. This made many problems magically disappear.

A few things that should be noted:
  1. CHECK TABLE will only update key statistics for MyISAM tables
  2. , for InnoDB you must use OPTIMIZE TABLE.
  3. If you use OPTIMIZE TABLE to update the key statistics you should be aware that OPTIMIZE TABLE does other (good) things as well, so it usually takes much longer and it locks your table while doing it. Also, for InnoDB it actually re-builds the table using ALTER TABLE.
  4. The CHECK TABLE have several options. Only the MEDIUM (the default) or EXTENDED update the key statistics. (i think).
So, if your indexes stop working, or just mis-behave, try to check them.

Friday, February 20, 2009

Vista fast user switching disappeared

My fast user switching option in Vista Home machine suddenly disappeared. I remember it was there, then it was not. I don't know why this happened, but here is how to fix it.

A rough translation:

Run regexit.exe
Go to HKEY_LOCAL_MACHINE > SOFTWARE > Microsoft > Windows > CurrentVersion > Policies > System
Look for a key named HideFastUserSwitching. If it exists, change it's value to 0 (zero). If it does not exist create it as a DWORD and set it to zero.
You might need to log off before the change takes effect.

Sunday, February 8, 2009

utf8 and hebrew in tomcat

In tomcat 5.0 and above, if your UTF-8 request parameters are received as gibberish you might need to do the following:

In your server.xml add the URIEncoding="UTF-8" and useBodyEncodingForURI="true" to the Connector tag(s):


redirectPort="8443" />

This should make GET requests work properly.

For some reason the above does not work for POST requests. If you ask the tomcat people they'll mumble something about W3C, RFC, and RTFM. The short way to have this work for POST requests is to write a small filter to set the request encoding properly. We are using something similar to this:

package com.realcommerce.filters;
import javax.servlet.*;
public class RequestEncodingFilter implements Filter {
public void init(FilterConfig filterConfig) throws ServletException {
//Do Nothing
public void destroy() {
//Do Nothing
public void doFilter(ServletRequest request,ServletResponse response,
FilterChain chain) throws IOException, ServletException
chain.doFilter(request, response);

This made POST requests pass Hebrew (or any UTF-8) parameters properly.

Tuesday, January 27, 2009

slow lstat, slow php, slow drupal

If you are short on time go to bottom line.

Ok, the lstat() and stat() system calls system call are not really slow per se. But take a look at an apache strace -r dump from a drupal installation I was testing:

[user@host ~]# strace -o strace.load -r -s 256 -p <PID>
0.000081 getcwd("/mnt/var/www/html/drupal", 4096) = 32
0.000048 lstat("/mnt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.000805 lstat("/mnt/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.001596 lstat("/mnt/var/www", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.000104 lstat("/mnt/var/www/html", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
0.000362 lstat("/mnt/var/www/html/drupal", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0
0.000804 lstat("/mnt/var/www/html/drupal/sites", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.057575 lstat("/mnt/var/www/html/drupal/sites/all", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.000297 lstat("/mnt/var/www/html/drupal/sites/all/modules", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.014762 lstat("/mnt/var/www/html/drupal/sites/all/modules/Internet", {st_mode=S_IFDIR|0755, st_size=4096,
0.000214 lstat("/mnt/var/www/html/drupal/sites/all/modules/Internet/date", {st_mode=S_IFDIR|0755, st_size=4
0.000112 lstat("/mnt/var/www/html/drupal/sites/all/modules/Internet/date/date.module", {st_mode=S_IFREG|064
0.010816 open("/mnt/var/www/html/drupal/sites/all/modules/Internet/date/date.module", O_RDONLY) = 20
0.000835 fstat(20, {st_mode=S_IFREG|0644, st_size=44118, ...}) = 0
0.000069 lseek(20, 0, SEEK_CUR) = 0
0.000063 stat("././sites/all/modules/Internet/date/date.module", {st_mode=S_IFREG|0644, st_size=44118, ...}) = 0
0.000159 close(20) = 0

As you can see, for every file included in the system there are around 10-15 lstat calls. this means that if we have ~200 files included we have ~2500 lstat calls per page. A simple sum on the strace results gave me ~8 seconds per page of lstat time, which was over a third of my test page time. see?

[user@host ~]# cat strace.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d |awk '{printf $2 "+"}' | sed s/.$/\\n/ | bc
[user@host ~]# cat strace.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d | fgrep lstat | awk '{printf $2 "+"}' | sed s/.$/\\n/ | bc
if you really want to understand this shell script let me know. It deserves a post by itself

Now we have 3 options: a) make lstat() faster. b) make php stop doing all those lstats(). c) make drupal include less files.

Well, (a) is not an option as our infrastructure team (aka sysadmins) insist there is nothing wrong with the filesystem and nothing can be done to enhance it's performance. Since I gave up being a sysadmin when it was still call 'sysadmin' it would be hard for me to prove them wrong (especially since they might be are right).

As (c) would require us to refactor most of our code in a very very 'not drupal way' and create very large php files, I was hoping to find a better solution.

By a rare chance of good fortune i came to learn about realpath_cache_size. Once I changed the foolish default to a reasonable 1M in php.ini, I lost most of the lstat() calls and over 10 seconds of processing time.

[user@host ~]# cat strace.2.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d |awk '{printf $2 "+"}' | sed s/.$/\\n/ | bc
[user@host ~]# cat strace.2.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d | fgrep lstat | awk '{printf $2 "+"}' | sed s/.$/\\n/ | bc

Here is what to do in php.ini

; ...php.ini...
; Determines the size of the realpath cache to be used by PHP. This value should
; be increased on systems where PHP opens many files to reflect the quantity of
; the file operations performed.

; Duration of time, in seconds for which to cache realpath information for a given
; file or directory. For systems with rarely changing files, consider increasing this
; value.

For normal pages under normal load I was able to get a 10%-25% performance improvment. I think that's quite a bit for a configuration change.

Thursday, December 11, 2008

Hebrew SEO: גוגל, עברית, וה"א הידיעה

נראה ש-google לא מתייחס נכון לה"א הידיעה.
אם נחפש לדוגמא הממשלה נקבל בתוצאות את "פורטל השרותים של ממשלת ישראל" אבל גוגל מסמן בבולד רק את המילה הממשלה ולא את המילים ממשלה או ממשלת שהן הטיות חוקיות לחלוטין.
בכיוון ההפוך, אם נחפש ממשלה נקבל תוצאות אחרות לגמרי, כאשר המילים ממשלת וממשלה מודגשות אבל המילה הממשלה אינה מוגדשת.
לפי ההבדלים בתוצאות החיפוש נראה שהבעיה קיימת לא רק בהדגשה אלא גם באינדוקס.
בדקתי גם בחיפוש לפי הילדים מול ילדים ומילים נוספות. הדבר נראה נכון גם לגבי live של מיקרוסופט (בלי לדבר על yahooשבכלל לא טרחו לתמוך במורפולוגיה עברית)
המסקנה,לצרכי אופטימיזציה במנועי החיפוש של גוגל כדאי להגדיר את מילות המפתח של האתר עם ובלי ה"א הידיעה.

Thursday, November 20, 2008

drupal memcache items disappearing from the cache

The scenario: you are using memcached as your Drupal cache system. You do a cache_set for some item. You do a cache_get for this item. The item is there. You do a cache_get again after a few seconds/minutes, the item is not there.

You might be experiencing this issue of the memcache module: In short it means that the memcache module flushes the entire cache cluster whenever someone does a wildcard cache_clear_all. Since it's very likely that all your bins are in a single cluster you get to a situation that whenever someone-on-the-other-side-of-the-world does something that is probably completely unrelated to you, your item will still get flushed from the cache.

Note that since flush does not activly removes the items from the cache but rather marks them as expired you will not see any memcached stats that will hint to this issue.

I have commented on the issue. Here are some highlights:
...we are currently using gaolei's patch which is fast albeit expensive. We are using it on a large production system that depends heavily on memcache and we did not see any problems yet. (thank you gaolei).

We have tried to implement a lock-add-unlock scenario such as was suggested in several comments but this will be a definite nightmare for high-traffic/high-update sites...

I would suggest letting the user choose which flush mechanism to use: the current one, or the salt one. I see no reason why the module's developers should decide for me. On a site with little updates and little memory i would prefer to have the whole memcache flush; On a site with many updates and tons of memory i'm willing to sacrifice space for the sake good performance. a simple memcache_flush_method variable would do the job just fine.

Monday, November 10, 2008

MYSQL Query Optimization: Avoiding ORs

So, you've probably heard that using ORs in your queries is heavy and should be avoided when possible. Here is a live example. We had the following query running on a medium sized table in MYSQL (around 200K rows):

SELECT count(*) FROM interactions p
(p.employer = 69 AND p.flag1 = 1)
(p.employee = 69 AND p.flag2 = 0)

Let's think of interactions as a table holding the interaction between employers and employees, and this query should tell the user the total number of interactions where user number 69 is involved.

The query looked pretty innoccent on an idle MYSQL server taking around 200ms, but when the server got busy the query execution time was reaching 5-6 seconds.

Altough we did have indexes on interactions.employer and interactions. employee doing EXPLAIN showed that MYSQL was not using them.

After some digging and fiddling and trying we ended up with:

SELECT count(*) as count FROM (
SELECT 1 FROM interactions p
WHERE (p.employer = 69 AND p.flag1 = 1)
SELECT 1 FROM privatemsg p
WHERE (p.employee = 69 AND p.flag2 = 0)
) t2

This query took on the busy server less then 250 milliseconds and less then 80ms on the idle server. Doing EXPLAIN showed that MYSQL was doing two queries, and using the correct index on each one.

After some more digging we noticed that if we EXPLAIN the original query on the idle server the optimizer occasionally converts it to a UNION, but for some reason this did not always happen and in any case took about twice the time then the UNION query.

To sum up the results here is a simple table

Query typeServer activityQuery Time
Using ORIdle200ms
Using ORBusy5500ms
Using UNIONIdle78ms
Using UNIONBusy189ms

Conclusion: always consider alternatives to OR, but make sure you check them well against real-time examples.

A Further Note: When timing queries in MYSQL alway use the optimizer hint /*! SQL_NO_CACHE */ to make sure you are not getting results from the query cache (if you have one set up).

Wednesday, September 10, 2008

Internet Explorer does not support http vary header

It was always assumed, but I just saw this quote from Microsoft:

Internet Explorer does not fully implement the VARY header per Requests for Comments (RFC) 2616. The Internet Explorer implementation of VARY is that it does not cache any data except for Vary-Useragent

Tuesday, September 9, 2008

Internet explorer cannot open the internet site: operation aborted

During an integration with a third party provider we were unfortunate enough to get the error
Internet explorer cannot open the internet site http://localhost: operation aborted
Yes, we have read and did move the 3rd-party's script to be just before the </BODY> tag but to no avail.

After a long and hard effort by this provider they pointed out that while the view-source indeed looked like the script is just before </BODY> when looking at the DOM itself (they used Dominspector, I used the IE Developer Toolbar) it showed that the script was inside a <DIV> element.

As it turns out there was some code on the page that wrote an unclosed <DIV>. This made IE "fill in the blanks" and guess (wrongly) where it should place the closing </DIV> Fixing this javascript made the error go away on most pages. But not all.

I found out that the broken pages used some JQuery plugins (tooltips, jqmodal, etc.) to produce fancy decorations and effects. These scripts attached to $document.ready and did $('body').appendTo(...). This effectively added a couple of items to the DOM between the 3rd-party script and the </BODY>. I am still not quite sure why this should cause IE to choke but since we have a tight schedule we simply changed those scripts to add their stuff to some other elements. This indeed solved out problem.

They funny (or maybe sad) thing was that on Firefox we did not get any error but the page simply disappeared. Chrome worked just fine.

Tuesday, August 26, 2008

Misplaced elements with position:relative

We had a situation where we had three <DIV> elements in a column and the middle element had content dynamically loaded into it after the page has completed loading. When the middle element finished loading, some of the content of the last element seemed to forget its parenthood and floated nicely over the second element.

elements with postion relative not placed properly after dynamic content loading

Naturally this only happened in Internet Explorer (what else).

After a lot of digging we noticed that the misplaced content of <DIV>#3 had a position:relative style (indicated as #4 in the image above). It is "well known" that IE does not handle relative positioning properly when the layout of the page change.

Luckily I chanced upon an article by Holly Bergevin et al. called On having layout which was enlightening. Understanding that IE will only respect the element's state if it have the hasLayout property quickly solved the problem.

In our specific case I used display:inline-block to force <DIV>#3 to have hasLayout set to true, which made IE respect the element's style and re-render the element's content (i.e. <DIV>#4) after re-positioning it. My only concern now is what effect this might have on rendering performance, but it will have to do for now.

Tip: you can use IE Developer Toolbar to check if an element hasLayout. It will show the hasLayout property as set to -1 if hasLayout is true.

Further credit is due to Ingo Chao who wrote relatively positioned parent and floated child – disappearance.

Monday, July 14, 2008

Absolute vs. Relative URLs and SEO

I've seen several places where people suggest that one should use absolute URLs ( to do internal linking instead of relative URLs (/page). Those people relate to a google article where it is quoted that
We also suggest you link to other pages of your site using absolute, rather than relative, links with the version of the domain you want to be indexed under. For instance, from your home page, rather than link to products.html, link to . And whenever possible, make sure that other sites are linking to you using the version of the domain name that you prefer

This seems to me a classic example of taking things out of context. If you read the entire article you can see that Ms. Fox is talking about a specific case where your site can be accessed by more then one domain name (e.g. and

There is nothing google ever wrote that i could find that say that absolute URLs are better if your site is only accessed by one domain name.

There is one exception i can think of: if your domain is "" for example and you do use absolute URLs, then the word "coolstuff" will appear in your pages alot. This might be something that may boost your ranking with regards the the word "coolstuff". But this is just a guess.

More reading: Google Canonicalization Problems? - Crawling, indexing, and ranking | Google Groups

Please note: this is my personal opinion. I've had at least two SEO experts that claim that absolute URLs will give better performance on google ranking. Since I'm strong-headed I will hold to my opinion until I'll be proven (with numbers) otherwise. Feel free to comment with supporting or conflicting opinions and data.

Sunday, July 6, 2008

Save binary file in Tcl under V6

This issue pops up every couple of years. It's nothing new, but still worth documenting.

The Challenge: To save a posted file to a file system, not using SUBMIT_STATIC_FILE.

The Problem: Vignette V6 pre-process the form submitted data so that any <input type="file"> are encoded as an hex string. If you ERROR_TRACE the variable for the file, you will see a string that looks like 0x1D00ABADD1D9....

The Solution: binary format to the rescue. and with a bit of trimming, you get it all in a few lines:

proc save_files { } {
#FileData,FileExtention,Field1 are posted from the <form>

set filename [generate_filename]
set bin_filename "${filename}.[SHOW FileExtention]"
set xml_filename "${filename}.xml"

### This will save a binary file.
set file [open $bin_filename "w"]
fconfigure $file -translation binary
puts -nonewline $file [binary format H* [string range [SHOW FileData] 2 end-1]]
close $file

### This will save a utf-8 xml (text) file
set file [open $xml_filename "w"]
fconfigure $file -encoding "utf-8"
puts $file "<?xml version='1.0' encoding='utf-8' ?>"
puts $file "<formdata>"
puts $file "<Field1>[HTML_ESCAPE [SHOW Field1]]</Field1>"
#...more fields...
$puts $file "<Attachment>$bin_filename</Attachment>"
puts $file "</formdata>"
close $file

* Remember to use H* and not h*.
* The example above also saves a text file in utf-8 with extra data from the form
* Make sure your form is enctype=multipart/form-data
* This will not work in Storyserver 4.2.

Thursday, June 5, 2008

Sunday, June 1, 2008

Apache Rewrites and SetEnv

It seems that variables set with Apache's SetEnv (and hence SetEnvIf and BrowserMatch) are being ignored by RewriteRule and RewriteCond. The rumor has it to be a design issue. The only way i could find around this problem is to use the [E= rewrite rule directive.

#Use RewriteRule & RewriteCond as a replacment for SetEnv and BrowserMatch
#SetEnv browser="-ie"
RewriteRule ^(.*)$ - [E=browser:-ie]
#BrowserMatch "IE7" browser="-ie7"
RewriteCond %{HTTP_USER_AGENT} "MSIE 7"
RewriteRule ^(.*)$ - [E=browser:-ie7]
#BrowserMatch "Firefox" browser="-moz"
RewriteCond %{HTTP_USER_AGENT} "Firefox"
RewriteRule ^(.*)$ - [E=browser:-moz]

RewriteCond %{DOCUMENT_ROOT}/cache/index%{ENV:browser}.html -f
RewriteRule ^(.*)$ cache/index%{ENV:browser}.html [L]

PS. If you need to debug rewrite rules you can add the following lines to your httpd.conf:
RewriteLog logs/rewrite.log
RewriteLogLevel 5

Wednesday, May 14, 2008

Annoying FireBug Bug

I came across an annoying bug in firebug. If there is a global variable with the same name as a local variable then firebug does not correctly show the value of the local variable in the tooltips and the watch window.

The most annoying thing is that it has been reported several months ago, but still no fix.

UPDATE: Confirmed Fixed in 1.2 on FF3.

Sunday, March 23, 2008

Drupal SMTP Authentication with PEAR::Mail

If you want to use smtp authentication with Drupal you can use PEAR::Mail with drupal_mail_wrapper.

  1. Install PEAR::Mail. On linux you can just run the following command as root:
     pear install --alldeps Mail

  2. Create a new file, say includes/smtpmail/ and paste the following code into it:

    require_once 'Mail.php';
    require_once 'PEAR.php';

    function drupal_mail_wrapper($mailkey, $to, $subject, $body, $from, $headers) {
    global $smtpmail;

    $headers['From'] = $from;
    if ( empty($headers['To'])) {
    $headers['To'] = $to;
    $headers['Subject'] = $subject;

    $recipients = $to;
    $mail_object =&Mail::factory('smtp', $smtpmail);
    $output = $mail_object->send($recipients, $headers, $body);
    return $output;

  3. Edit your settings.php and add the following at the end:

    $conf = array(
    'smtp_library' => 'includes/smtpmail/'
    global $smtpmail;
    $smtpmail= array();
    $smtpmail["host"] = '';
    $smtpmail["auth"] = TRUE;
    $smtpmail["username"] = 'your_user_name';
    $smtpmail["password"] = 'your_password';

From now on every call to drupal_mail will go through your drupal_mail_wrapper and will be sent using PEAR::Mail with SMPT authentication.

I'm sure some more serious drupaler would have made this a nice module, with configuration screens and the lot, but I'm too lazy today.

Monday, March 17, 2008

"The system cannot execute the specified program" Error

You are trying to run some program on Windows (such as apache.exe or htpasswd.exe) and you are getting "The system cannot execute the specified program" error. This usually means that the program you are trying to run was compiled against DLLs that are not on your system. You should use Dependency Walker to see which DLLs are missing, and then install them.

The Apache 2.x binary windows distribution, specifically, was compiled against the Visual Studio 2005 re-distributable package, which you can download from microsoft.

Further reading (1) and (2)

Edit (Nov 2011):
Following this comment (by Annonymous) please note that you may need to download and use the 2008 package and not the 2005 one: The official download for the Microsoft Visual C++ 2008 SP1 Redistributable Package (x86) or The official download for the Microsoft Visual C++ 2008 Redistributable Package (x64)

Sunday, February 17, 2008

Remote Root Access to MySql

To help me remember:

mysql -u root
mysql> SET PASSWORD FOR 'ROOT'@'LOCALHOST" = PASSWORD('new_password');
mysql> exit;

Source: ben robison :: Howto: Remote Root Access to MySql

Wednesday, January 16, 2008

phpize: command not found

If you ever get phpize: command not found error when trying to install a PEAR package on a linux system, this is probably becuase the php-devel package is not installed.

Uninstall Google Desktop Gadget

In order to uninstall any Google Desktop gadget you can do the following:
1. Close Google Desktop
2. Open 'My Documents' folder and locate and open 'My Google Gadgets' folder in it
3. Find the gadget you want to remove and delete it.
4. Open Google Desktop again.

(source: uninstall RSStoSpeech Google Desktop gadget - RSS To Speech Gadget | Google Groups)

Thursday, January 3, 2008

document.body.scrollTop in IE

It seems that document.body.scrollTop does not work in IE6 if the document's doctype is defined as
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
To find out the scrolling offset in a way that works for both DTD3 and DTD4.01 you can use
Same goes for scrollLeft.
See more at document.body.scrollTop in IE

Sunday, December 23, 2007

Blog Archive Carousel Widget

I've played around with blogspots' widget system and came up with a nice blog archive carousel widget. The widget is based on the standard blog archive widget and jcarousel, a jquery plugin.
DISCLAIMER: Use at your own discretion.
How to use:

  • Go to your blog's Template -> Edit HTML.

  • Download Full Template for backup and save it somewhere you'll remember.

  • Go go Template -> Page elements and add a blog archive widget, save the page.

  • Go back to Template -> Edit HTML.

  • Make sure Expand Widget Templates is not checked.

  • Just below the <data:blog.pagetitle/> add the code below

    <!-- archive carousel -->
    <script src=''/>
    <script src=''/>
    <script type='text/javascript'>
    jQuery(document).ready(function() {
    jQuery('#mycarousel').jcarousel({ vertical: true, scroll: 3}); });
    <!-- //archive carousel -->

  • At the end of the <skin> section, just above the ]]></b:skin>add the code below

    /* new archive */
    UL.posts LI { font-size:80%; width:170px; padding-right:2px; border-bottom:1px solid silver;}
    .jcarousel-container { position: relative; }
    .jcarousel-clip { z-index: 2; padding: 0; margin: 0; overflow: hidden; position: relative; }
    .jcarousel-list { z-index: 1; overflow: hidden; position: relative; top: 0; left: 0; margin: 0; padding: 0; }
    .jcarousel-item { float: left; list-style: none; width: 170px; xheight: 18px; }
    .jcarousel-next { z-index: 3; display: none; }
    .jcarousel-prev { z-index: 3; display: none; }
    .jcarousel-skin-tango.jcarousel-container { -moz-border-radius: 10px; background: #F0F6F9; border: 1px solid #346F97; }
    .jcarousel-skin-tango.jcarousel-container-vertical { width: 170px; height: 170px; padding: 40px 20px; }
    .jcarousel-skin-tango .jcarousel-clip-vertical { width: 190px; height: 170px; }
    .jcarousel-skin-tango .jcarousel-item { width: 170px; xheight: 18px; }
    .jcarousel-skin-tango .jcarousel-item-vertical { margin-bottom: 10px; }
    .jcarousel-skin-tango .jcarousel-item-placeholder { background: #fff; color: #000; }
    .jcarousel-skin-tango .jcarousel-next-vertical { position: absolute; bottom: 5px; left: 99px; width: 32px; height: 32px; cursor: pointer; background: transparent url( no-repeat 0 0; }
    .jcarousel-skin-tango .jcarousel-next-vertical:hover { background-position: 0 -32px; }
    .jcarousel-skin-tango .jcarousel-next-vertical:active { background-position: 0 -64px; }
    .jcarousel-skin-tango .jcarousel-next-disabled-vertical,
    .jcarousel-skin-tango .jcarousel-next-disabled-vertical:hover,
    .jcarousel-skin-tango .jcarousel-next-disabled-vertical:active { cursor: default; background-position: 0 -96px; }
    .jcarousel-skin-tango .jcarousel-prev-vertical { position: absolute; top: 5px; left: 99px; width: 32px; height: 32px; cursor: pointer; background: transparent url( no-repeat 0 0; }
    .jcarousel-skin-tango .jcarousel-prev-vertical:hover { background-position: 0 -32px; }
    .jcarousel-skin-tango .jcarousel-prev-vertical:active { background-position: 0 -64px; }
    .jcarousel-skin-tango .jcarousel-prev-disabled-vertical,
    .jcarousel-skin-tango .jcarousel-prev-disabled-vertical:hover,
    .jcarousel-skin-tango .jcarousel-prev-disabled-vertical:active { cursor: default; background-position: 0 -96px; }

  • find the
    <b:widget id="'BlogArchive1'" locked="'false'" title="'Blog" type="'BlogArchive'/">
    and replace this line with the following code

    <b:widget id='BlogArchiveCarousel' locked='false' title='Blog Archive' type='BlogArchive'>
    <b:includable id='toggle' var='interval'>
    <b:if cond='data:interval.toggleId'>
    <b:if cond='data:interval.expclass == "expanded"'>
    <a class='toggle' expr:href='data:widget.actionUrl + "&amp;action=toggle" + "&amp;dir=close&amp;toggle=" + data:interval.toggleId + "&amp;toggleopen=" + data:toggleopen'>
    <span class='zippy toggle-open'>&#9660; </span>
    <a class='toggle' expr:href='data:widget.actionUrl + "&amp;action=toggle" + "&amp;dir=open&amp;toggle=" + data:interval.toggleId + "&amp;toggleopen=" + data:toggleopen'>
    <span class='zippy'>
    <b:if cond='data:blog.languageDirection == "rtl"'>
    <b:includable id='interval' var='intervalData'>
    <b:loop values='data:intervalData' var='i'>
    <b:if cond=''>
    <b:include data='' name='interval'/>
    <b:if cond='data:i.posts'>
    <b:include data='i.posts' name='posts'/>
    <b:includable id='menu' var='data'>
    <select expr:id='data:widget.instanceId + "_ArchiveMenu"'>
    <option value=''><data:title/></option>
    <b:loop values='data:data' var='i'>
    <option expr:value='data:i.url'><> (<>)</option>
    <b:includable id='flat' var='data'>
    <b:loop values='data:data' var='i'>
    <li class='archivedate'>
    <a expr:href='data:i.url'><></a> (<>)
    <b:includable id='posts' var='posts'>
    <b:loop values='data:posts' var='i'>
    <li>&amp;nbsp;<a expr:href='data:i.url'><data:i.title/></a></li>
    <b:includable id='main'>
    <b:if cond='data:title'>
    <div class='widget-content'>
    <div id='ArchiveList'>
    <div expr:id='data:widget.instanceId + "_ArchiveList"'>
    <ul class='jcarousel-skin-tango posts' id='mycarousel'>
    <b:include data='data' name='interval'/>
    <b:include name='quickedit'/>

  • Preview the template

  • If all looks well, you can save your template. You might get prompted to confirm that BlogArchive1 will be deleted. Allow it (we changed it's name to "BlogArchiveCarousel")

Tuesday, December 18, 2007

MySQL 5.1 Clustering Might Just Actually Work

As I previously wrote mysql 5.0 clustering was a big issue for us since all data as placed in memory. It seems that mysql 5.1 clustering is going to be much better, including the ability to store non-index columns on disk (indexes and indexed columns are still stored in-memory), do (limited) cluster-to-cluster replication, and better varchar support for memory columns. Pitty it's only release candidate at the moment. can't wait for the GA to come out.

Monday, December 17, 2007

Tomcat Cannot assign requested address

I had a very strange error today when I tried to startup tomcat on some linux box that used to work. Since google was no help I figured I should document this somewhere.

Tomcat startup gave the notorious Cannot assign requested address. This usually means that another process is listening on the same port on this machine. I started with ps -ef grep java to make sure that no other tomcat process was running on the machine. Nothing.

I suspected that maybe some other web server was holding the port, but netstat -an showed that nothing binds this address. Moreover, telnet localhost 80 showed that no one is listening.

Looking at the server.xml I found out that the connector was binding to the hostname "demoserver" on port 80.

<Connector className="org.apache.coyote.tomcat4.CoyoteConnector"
address="demoserver" port="80"
minProcessors="5" maxProcessors="75"
enableLookups="true" redirectPort="8443"
acceptCount="100" debug="0" connectionTimeout="20000"
useURIValidationHack="false" disableUploadTimeout="true" />

trying to ping demoserver gave me errors, but nslookup demoserver gave me the right ip address.

After a few minutes I noticed that some clown added demoserver to /etc/hosts with the wrong ip address. Fixing /etc/hosts to have the right ip address for demoserver made tomcat work smooth and nice.

Sunday, December 9, 2007

PHP Allowed memory size of 8388608 bytes exhausted

If you try to install PHP PECL package and get an "Allowed memory size of 8388608 bytes exhausted" you should know that pecl ignores php.ini memory_limit directive. As suggested in
this post, you should use pear instead of pecl to install you package. For example, instead of doing

[root@hostname tmp]# pecl -v install APC

you should do

[root@hostname tmp]# pecl -v download APC
[root@hostname tmp]# pear -v install APC-3.0.15.tgz

Wednesday, November 21, 2007

Java TTL Cache, the Quick and Dirty Way

There are many caching systems in java. I really like some of them (especially Ian Schumacher's LRU Cache). But sometimes you just to drop a quick few lines to get something in a local cache. Here is one way of doing it:

private static Map urlItemCache = Collections.synchronizedMap(new HashMap());
private static Map urlItemCacheTtl = Collections.synchronizedMap(new HashMap());
private static final long TTL=1000*60*60;//one hour in milliseconds

public static String findItemByUrl( String url) {
String item="";
Long now=Long.valueOf(Calendar.getInstance().getTimeInMillis());

//try to get the item from the local cache
item = urlItemCache.get(url);
if ( !isEmpty(item)) {
//urlItemCacheTtl.get(url) is guaranteed to have a value, since it's added at the same time urlItemCache is.
if (now.longValue() - urlItemCacheTtl.get(url).longValue() < TTL ) {
urlItemCacheTtl.put(url, now); //see comment below
return item;
} else {
//not found in cache
//... do you application magic here to get the item ...
if ( !isEmpty(item) ) {
urlItemCache.put(url, item); //add to local cache
urlItemCacheTtl.put(url, now);
return item;
} //else
return null;

Wednesday, September 26, 2007

Reload webapp from ant script

A co-worker asked me how he can reload a web-application from an ant script. GDS was very helpfull finding the line in an old ant script I used to use. In the example below you might want to define $project.output

<property name="project.output" value="WEB-INF/classes"/>
<!-- reload the current webapp to refresh the classes that we just changed -->
<concat><fileset dir="${project.output}/log" includes="reload.log"/></concat>

Monday, September 24, 2007

RequestDispatcher.forward() and filters

If you have read this post then you probably know that if you use RequestDispatcher.forward() (or RequestDispatcher.include()) the filter chain does not get re-called for the forwarded servlet/jsp. However, if you read more closely you see that Craig was talking about the 2.3 servlet spec. Since we are using tomcat 5.5 which conforms to the 2.4 spec, you should know that there's a new <dispatcher> element in the deployment descriptor with possible values REQUEST, FORWARD, INCLUDE, and ERROR. You can add any number of entries to a <filter-mapping> tag and individually decide which filter gets called when. An Example:



Simpler GUI for Jad, The Java Decompiler from As I Learn

This is definitely the most useful two-liner I've seen in a while:
Simpler GUI for Jad, The Java Decompiler.

I naturally customized it to:

jad -o -s java -lnc -d%TEMP% %1
"C:\Program Files\TextPad 5\TextPad.exe" %TEMP%.\

Changed extension to java so that textpad does syntax highlighting; added -lnc so that jad outputs the original line numbers as comments and I can trace the exceptions I get

(after a few questions from people i'm editing this post:)
You will need to download jad and extract the exeutable to somewhere on your path, say C:\windows (or c:\winnt). Then create a new .cmd file, paste the above into the .cmd file, and put it somewhere. Then rightclick any .class file you can find, choose "Open With..."->"Choose Program"->"Browse..." and find the cmd file you created (don't forget to check the "Always use this program"). This way, any .class you'll double-click will decompile and open in textpad.

Monday, September 17, 2007

Using java regular expressions to match groups

It's been demonstrated over and over again, but I always seem to forget the "simple example". So here it is:

import java.util.regex.Matcher;
import java.util.regex.Pattern;
*Going to match a URL that looks like this
* /url/prefix/--sitename--/menu.--menuid--/?param=--itemid--
* and change it into
* /url/prefix/--sitename--/--menuname--/--itemname--
String expr="(/url/prefix/[a-zA-Z0-9_-]+)/menu\\.([a-zA-Z0-9_-]+)/\\?param=([a-zA-Z0-9_-]+)";
Pattern p = Pattern.compile(expr);
Matcher m = p.matcher(uri);
if ( m.matches() ) {
menuitem =;
guid =;
if ( guid != null && prefix != null && menuitem != null ) {

Thursday, September 6, 2007

Does GDS slows my hard disk?

I've a encountred a strange problems that when Google Desktop Search is running on my PC (which it almost always is), my hard disk's read speed is about 1/6th of the normal speed. I'll contact google to see if this is a known issue. I'm using several HD benchmark tools to test the speed, most notably Simpli Software's Hd Tach.

Here are screenshots of my disk's speed with and without GDS running.

Avoding browser history when changing iframe src

We ran into a problem where we needed to have an iframe who's source needed to be changed dynamically by a javascript based on some user data. The issue was that Internet Explorer saved the iframe's old url in the history. The simple solution is to dynamically write the iframe with it's real src instead of having an iframe on the page and then changing it's src.

You do it like this:

var iframeHeaderCell = document.getElementById('wheretoputheiframe');
var dynamicURL = 'http://...' //your url
var iframeHeader = document.createElement('IFRAME'); = 'iframeHeader';
iframeHeader.src = dynamicURL ;
iframeHeader.width = ...;
iframeHeader.height = ...;
iframeHeader.scrolling = 'no';
iframeHeader.frameBorder = 0;
iframeHeader.align = 'center';
iframeHeader.marginwidth = 0;
iframeHeader.marginheight = 0;
iframeHeader.hspace = 0;
iframeHeader.vspace = 0;

Elementry, my dear Watson.

Monday, August 13, 2007

Updateing your docroot from subversion

We need to update our integration environment from svn every once in a while. At the moment a manual process is required since our svn gets commited into even if not everything is really stable and we only want to update the integration env when we feel the system is stable. We could do it manually by ssh-ing into the machine, but i hate people logging into my box to do stuff that should have a web interface, and we will probably want to extend that interface in the future.

Here is what I did:

First, manually do an svn checkout in the docroot:

su -
cd /var/www/html
svn co --username myusername svn://mysvnhost/svn/repositories/repository-name/projectname projectname
chown -R apache:apache projectname

Second, you will want a small PHP script to do all the work for you, so you will not have to ssh to machine all the time:

$log = "ACTION LOG\n---------------\n\n";
$log .= shell_exec("sudo svn up --username $username --password $password $docroot/$project");
echo "<pre>$log</pre>";

The thing is that since apache does not change it's user properly, svn cannot be properly run as apache, so will need the sudo there.

Third, in order for all of this to work, you will need to setup /etc/sudoers properly so go ahead and add this to it:

apache ALL=NOPASSWD:/usr/bin/svn

This will of course grow into something much more elaborate, that allows you to select projects, update on schedule, etc.

note that if you do what i specified above, then every update from subversion will reset the owner for the files under the docroot to root. This is generally not a good idea, so i suggest you put the svn update line in a shell script and add a "chown -R apache:apache projectname" to that script. then you can just sudo from the php file.

Tuesday, July 31, 2007

If you need to monitor mysql proccesslist, dump the following to a shell script and chmod 755. (ofcourse you will need to add passwords etc. if you have any;)


watch 'mysql<<EOF
show processlist;

(Control+C will let you out)

Two webapps using shared jar - ClassLoader problem

Yeah, yeah, I know it's obvious, but i thought i might be smarter. I am not:

Charles R Caldarale Writes:

[When using Tomcat,] classes from different webapps cannot access each other - no ifs, ands, or buts. Anything that is to be accessed from more than one webapp must be put in a common location [such as tomcat/shared].


Monday, June 18, 2007

Internet Explorer CSS Limit

We just came across a bug that makes IE choke when it needs to handle over 30 <style> objects. If you thought of using a single style object and then @import the other files, it won't work either.

You might want to ask why we need over 30 style objects. Well, since each drupal module tends to add it's own CSS (using drupal_add_css) we very quickly got to 22 CSS's being included. From there the road to 30-32 was quite trivial. Guess we'll have to use Drupal's CSS/JS aggragator to solve this.

Sunday, June 10, 2007

Calcualting Average in UNIX Shell Script

I grabbed a piece of code to calculate an average for data in shell. It didn't work so I had to modify it. This is the result.

while read x
sum=`echo $sum + $x | bc`
if [ "$?" -eq "0" ]; then
n=`expr $n + 1`
echo "scale=2;$sum/$n"| bc

Thursday, June 7, 2007

Server Header in Apache Tomcat

Altough you cannot disable the header, you can change it to anything you like by adding a "server" attribute to the connector tag in server.xml

port="80" address="localhost"