Moved to Medium

I'm putting all new stuff on Medium at https://medium.com/@nirlevy
See you there...

Native Fullstack Developer for DayZz

מפתח native fullstack

ל- dayzz דרוש/ה מפתח/ת מובייל וסרבר לפיתוח מוצר חדשני בתחום ה- health-tech. הזדמנות להתחיל מסע מאתגר משלביו הראשונים ולייצר פתרונות אשר ישפרו את חייהם של מיליוני אנשים.

דרישות טכניות

·      לפחות 3 שנים בפיתוח בעולמות הווב/מובייל כולל פיתוח javascript ברמה גבוהה

·      לפחות שנה בפיתוח אינטנסיבי ב-react, react-native או native android/ios

·      ניסיון בפיתוח צד שרת מבוסס node.js או django/python ובמיוחד פיתוח API

·      ניסיון בפיתוח מעל בסיס נתונים nosql כגון mongodb

·      ניסיון בעבודה עם AWS או  Google Cloud

 

את מי אנחנו מחפשים?

·      אדם עם ראש פתוח שיכול לחשוב יצירתית ולא לפחד להביע את דעתו

·      מישהו (או מישהי) שיודעים לעבוד עם צוותי מוצר באווירה דינאמית

·      חשיבה מוצרית מוכוונת לקוח, והבנה של יסודות חווית משתמש

מה תעשו אצלנו?

·      פיתוח react native בצד הקליינט, ויישום של חווית משתמש מעולה

·      פיתוח ממשקי api ב- node.js בצד השרת, בעבודה מול mongodb

·      התממשקות לרכיבים צד ג׳ כגון firebase ושירותי ענן נוספים

·      פיתוח ממשקים לרכיבי חומרה חיצוניים (wearables ו-accessories)

קורות חיים ופרטים נוספים: jobs@dayzz.com

Timing django/python function calls

Doing some optimization for SWITCH.co.il I've come across a handy snippet to time the execution of python/django functions. Since I was doing deep-dive checking I also wanted to know the stack of the executed function. Figured someone (me) would want to use it in the future, so I'm adding it here. Not the prettiest of code, but it works.

import time
import inspect                                                
def timeme(method):
    def time_wrapper(*args, **kw):
        startTime = int(round(time.time() * 1000))
        result = method(*args, **kw)
        endTime = int(round(time.time() * 1000))
        stack=[]  
        for i in inspect.stack():
            if i[3] == 'time_wrapper': continue 
            stack.append(i[3])
        print 'Time(ms): ', ('%04d')%(endTime - startTime), method.__name__, u'->' ,u'->'.join(reversed(stack[0:5]))
        #only show 5 top stack
        return result
    return time_wrapper

#usage example
@timeme
def some_function(x,y):
    do_something
    return something

RTL Bootstrap CSS Convertor

Bootstrap.css is great, Bootstrap.css 3 is better. Alas, Neither have an RTL version. There are a couple of not-maintained gits floating around the web for bootstrap 2 but non for bootstrap 3

To convert bootstrap 3 to RTL you can use https://github.com/ded/R2

If you want a rather stable, working, version of RTL bootstrap.css you can use the one from my repository at https://bitbucket.org/levynir/bootstrap3-rtl. Even if you decide to brew your own version, I recommend that you read the README on this bitbucket as there are some caveats to RTLing bootstrap.

Setting I18N Hebrew in Django 1.5

It should be simple. But the docs are not very clear. So here is how to setup your Django site to be in Hebrew (or any other language).

The need: Make my site work in Hebrew, without taking into account Browser's Accept Language and all this stuff.

Ingredients: Working Django 1.5 project on Linux, common sense

Steps:


Install GNU gettext (even if you think you have it already)

sudo apt-get install gettext

Change in projectname/settings.py

# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
# LANGUAGE_CODE = 'en-us'
# This will force this installation to be in Hebrew
LANGUAGE_CODE = 'he'

#...

# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True

# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale.
USE_L10N = True

# If you set this to False, Django will not use timezone-aware datetimes.
USE_TZ = True

Make sure your MIDDLEWARE_CLASSES do not have LocaleMiddleware. If you do have LocaleMiddleware than Django will try to guess the right locale from the request context, which we don't want at the moment.

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    #Comment out LocaleMiddleware if it's there to keep in the same locale all the time
    #'django.middleware.locale.LocaleMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    # Uncomment the next line for simple clickjacking protection:
    # 'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

You may also need to add django.core.context_processors.i18n to your TEMPLATE_CONTEXT_PROCESSORS. (Don't do that right now, wait and see if everything is working and only add this if stuff is not working)

#add this at the end of settings.py if I18N is not working
from django.conf import global_settings
TEMPLATE_CONTEXT_PROCESSORS = global_settings.TEMPLATE_CONTEXT_PROCESSORS + (
    ".context_processors.site",
    "django.core.context_processors.i18n",
)

In your python code wrap translatable texts with ugettext_lazy or ugettext (you can and should alias them to some conventional shortcut (I use _t for ugettext and _ for ugettext_lazy, for example

from django.utils.translation import ugettext_lazy  as _
from django.utils.translation import ugettext  as _t
class Shirt(models.Model):
    COLOR_TYPES = (
                    ('B', _('Blue')),
                    ('G', _('Green')),
                    )
    
    ...
    color = models.CharField(max_length=1, choices=COLOR_TYPES)
    type(self):
        return _t('Shirt')

In your templates you need to use this (ugly and hard to read) method

{% load i18n %}
<!-- Make sure this LANGUAGE_CODE is correct: {{LANGUAGE_CODE}} -->
...
<h1>{%trans "Shirt Store"%}</h1>
...
or you can use {{_("Shirt Store")}} as an alternative method

Remember that you need to {% load i18n %} in every template that uses {%trans%}.

Now we need to create the tranlsation files themselves:

cd django-project
cd django-app
mkdir locale
django-admin.py makemessages -l he
vi locale/he/LC_MESSAGES/django.po
django-admin.py compilemessages

Rinse, repeat

Run your site and hope for the best.

Sources:

• http://stackoverflow.com/questions/1832709/django-how-to-make-translation-work
• https://docs.djangoproject.com/en/1.5/topics/i18n/translation/
• http://devdoodles.wordpress.com/2009/02/14/multi-language-support-in-a-django-project/
• http://stackoverflow.com/a/7633261/130304

"Type 'Newtonsoft.Json.Linq.JToken' is a recursive collection data contract" While Adding Service Reference in VS2012

Trying to add a service reference in Visual Studio 2012 to Bing Maps SOAP service I got the following error message:

Cannot import wsdl:portType
Detail: An exception was thrown while running a WSDL import extension: System.ServiceModel.Description.DataContractSerializerMessageContractImporter
Error: Type 'Newtonsoft.Json.Linq.JToken' is a recursive collection data contract
...

Following Getting “Recursive collection data contract” when referencing a WCF service with a slightly complex method and Visual Studio 2012 Error Reference.svcmap throwing Error I've had to change the "reuse types in referenced assemblies" from "all" to everything except Newtonsoft.JSON and RestSharp.

Might not be something to write home about, but I'm sure I'll forget where it is was in a year or so.

PS. This seems to be VS2012 only issue, VS2010 worked just fine.

Tech-Ed Eilat 2010

So happy to be presenting at Tech-Ed Eilat 2010.
See you all at the Herods Hotel, Tue 30/11 at 11:15 where we would share our lessons from the field on enterprise web content management with SharePoint 2010.

You can read more about our session at the realcommerce web site.

IIS JNLP 404 Problem

If your IIS 6 returns a 404 Not Found when you request a Java Web Start jnlp file, then you probably need to add the jnlp mime-type to the IIS configuration.

IIS Settings -> Right click the machine -> mime-types -> new Extension: .jnlp MIME Type: application/x-java-jnlp-file

How to do rewrites in an IIS ISAPI

Note to self: If ever you need to do a rewrite in an IIS ISAPI Filter you just need to re-set the pHeaderInfo's url header. Something like this:

 
pHeaderInfo->GetHeader(pCtxt->m_pFC,
              "url",strUrl.GetBuffer(dwUrlSize+1),&dwUrlSize);
strUrl.ReleaseBuffer(); 
strNewUrl = GET_REWRITE_FOR_URL(strUrl); 
if ( strNewUrl.IsEmpty() == FALSE ) {  
    pHeaderInfo->SetHeader(pCtxt->m_pFC, 
              "url", (LPTSTR)(LPCTSTR)strNewUrl);  
} 
return SF_STATUS_REQ_NEXT_NOTIFICATION; 

This is working in IIS4 and above.

Note that if you really need a good re-write/redirect filter for IIS you should look at IIRF ,but since it's not working for IIS4 i had to do this manually

Bing, Google, Nutch and Canonical URLs

Google announced a few months back that they started supporting canonical urls. This is a great feature that we already adopted in a couple of sites (mostly to keep tracking codes from messing with our search engine results.

Unfortunelty, altough Microsoft announced they will support this feature in Live, this was not yet implemented in Bing as of now (Jun 2009).

Nutch, on the other hand, had this on thier TODO list for 1.1.

Also, Google Search Appliance (GSA), is currently assumed by all to not support this (although nobody really knows if it does).

More about canonical URLs:
http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html
http://searchengineland.com/canonical-tag-16537
http://janeandrobot.com/library/url-referrer-tracking

many group by strategies to select single rows per group

Xaprb gives a good explanation for selecting a single row from a group: How to select the first/least/max row per group in SQL at Xaprb.

apc futex_wait lockdown make your apache freeze over

We had the typical LAMP setup going on, with Drupal as the base CMS and APC for bytecode cache. We needed a good caching engine so I figured why not use APC's user cache. Well, we tried the APC Cache Drupal Module which, with minor fixes proved to work very nicely. That is, until we actually put this all thing on production.

The first thing we had was having our apache hang and not respond to any user requests. We susspected network issues, especially since netstat -na showed that all the apache processes were hanging on SYN_WAIT. However, since apache restart solved the issue i started to suspect this was something else.

To make a long (very long) story short, I got strace on our prod machines to find out that apache was either hanging on futex_lock(....FUTEXT_WAIT...) or doing infinte loops on the same functions.

To make even a longer story short, I got gdb installed on those machines and the backtrace clearly indicated that the locks were from APC user-cache calls.

We decided to abandon APC user-cache and switch to memcached which proved faster and had less lockdowns.

The funny thing is that when we talked about this over dinner the same evening a developer from another team just pointed me to this article by one of the APC leaders: (or something) How to Dismantle an APC Bomb which has been around for over a year. I am supprised and shocked that such a information is hidded so well and not mentioned anywere in the docs. Moreover, I went through the APC code again after reading this post (I went through it once when i started analysing the problem) and it seems that this is not even close to being resolved. there are no patches and no TODOs and nothing of the sort. From reading the code the entire user-cache needs a major re-write. What gives?

(this is a post i wrote a couple of months ago, never had time to finish it. Unfortunately this is still not fixed afaik)
EDIT (07/2009): http://pecl.php.net/bugs/bug.php?id=15179 reports this to be fixed. If anyone can confirm this please send me a note so that I could update this post

GA_googleAddAttr is not defined error

If you are using Google Ad Manager you can define custom attributes for targeting.

If you are getting a javascript error GA_googleAddAttr is not defined just make sure to put the call(s) to GA_googleAddAttr in a seperate <script> line, after the GS_googleEnableAllServices.

<script type="text/javascript">
GS_googleAddAdSenseService("ca-pub-0000000000"); 
GS_googleEnableAllServices();
</script>
<script type="text/javascript">
GA_googleAddAttr("ATTRB1", "whatever");
</script>
<script type="text/javascript">
GA_googleAddSlot("ca-pub-0000000000", "slotname_216x311");
</script>

V6 DECODE64 not working properly

It turns out that after all these years, V6's built-in DECODE64 and ENCODE64 actually do not work properly with some binary data.

proc COMPARE_BASE64 {} {
 set code  {2lIU1ZNw5BYvKi79j4L/+GGmjAQK7uiBQG7elDdKZcE=}
 set vgn_result [DECODE64 $code];         #VGN built-in function
 set tcl_result [::base64::decode $code]; #TclLib tcl-only implementation 
 return [join [list \\
   [BIN2HEX $vgn_result] \\
   [BIN2HEX $tcl_result] \\
   [string compare $vgn_result $tcl_result ] \\
  ] "\\r\\n"]
}
proc BIN2HEX { text }  { binary scan $text H* result; return $result }
COMPARE_BASE64

--- result ----
da52145370e4162f2a2efd8f82fff861a68c040aeee881406e94374a65c1
da5214d59370e4162f2a2efd8f82fff861a68c040aeee881406ede94374a65c1
1

If you need to encode/decode base64 for use in non-vignette systems make sure you use tcllib's base64.

This was tested on Vignette's V6 but I am sure it's true for Storyserver 4.2 and V5 as well.

MySQL indexes behaving badly

If you have a MySQL query that for no apparent reason stops working or no longer behaves as it used to, the first thing to do is EXPLAIN it. If the explain is unreasonable (for example, using different indexes then it used to or using an index but still scanning millions or rows) then you should try to CHECK TABLE.

One thing that I accidently found out was that CHECK TABLE also updates the key statistics. This made many problems magically disappear.

A few things that should be noted:

1. CHECK TABLE will only update key statistics for MyISAM tables
, for InnoDB you must use OPTIMIZE TABLE.

2. If you use OPTIMIZE TABLE to update the key statistics you should be aware that OPTIMIZE TABLE does other (good) things as well, so it usually takes much longer and it locks your table while doing it. Also, for InnoDB it actually re-builds the table using ALTER TABLE.
   
3. The CHECK TABLE have several options. Only the MEDIUM (the default) or EXTENDED update the key statistics. (i think).

So, if your indexes stop working, or just mis-behave, try to check them.

Vista fast user switching disappeared

My fast user switching option in Vista Home machine suddenly disappeared. I remember it was there, then it was not. I don't know why this happened, but here is how to fix it.

A rough translation:

Run regexit.exe
Go to HKEY_LOCAL_MACHINE > SOFTWARE > Microsoft > Windows > CurrentVersion > Policies > System
Look for a key named HideFastUserSwitching. If it exists, change it's value to 0 (zero). If it does not exist create it as a DWORD and set it to zero.
You might need to log off before the change takes effect.

utf8 and hebrew in tomcat

In tomcat 5.0 and above, if your UTF-8 request parameters are received as gibberish you might need to do the following:

In your server.xml add the URIEncoding="UTF-8" and useBodyEncodingForURI="true" to the Connector tag(s):

    <Connector 
     useBodyEncodingForURI="true" 
     URIEncoding="UTF-8" 
     acceptCount="100" 
     enableLookups="false" 
     maxSpareThreads="75" 
     maxThreads="150" 
     minSpareThreads="25" 
     port="8080" 
     redirectPort="8443" />

This should make GET requests work properly.

For some reason the above does not work for POST requests. If you ask the tomcat people they'll mumble something about W3C, RFC, and RTFM. The short way to have this work for POST requests is to write a small filter to set the request encoding properly. We are using something similar to this:

package com.realcommerce.filters;
import javax.servlet.*;
import java.io.*;
public class RequestEncodingFilter implements Filter  {
    public void init(FilterConfig filterConfig) throws ServletException {
         //Do Nothing
    }
    public void destroy() {
        //Do Nothing
    }
    public void doFilter(ServletRequest request,ServletResponse response,
        FilterChain chain) throws IOException, ServletException 
    {
        request.setCharacterEncoding("UTF-8");
        chain.doFilter(request, response);
    }
}

This made POST requests pass Hebrew (or any UTF-8) parameters properly.

slow lstat, slow php, slow drupal

If you are short on time go to bottom line.

Ok, the lstat() and stat() system calls system call are not really slow per se. But take a look at an apache strace -r dump from a drupal installation I was testing:

[user@host ~]# strace -o strace.load -r -s 256 -p <PID>
...
0.000081 getcwd("/mnt/var/www/html/drupal", 4096) = 32
0.000048 lstat("/mnt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.000805 lstat("/mnt/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.001596 lstat("/mnt/var/www", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.000104 lstat("/mnt/var/www/html", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
0.000362 lstat("/mnt/var/www/html/drupal", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0
0.000804 lstat("/mnt/var/www/html/drupal/sites", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.057575 lstat("/mnt/var/www/html/drupal/sites/all", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.000297 lstat("/mnt/var/www/html/drupal/sites/all/modules", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
0.014762 lstat("/mnt/var/www/html/drupal/sites/all/modules/Internet", {st_mode=S_IFDIR|0755, st_size=4096,
0.000214 lstat("/mnt/var/www/html/drupal/sites/all/modules/Internet/date", {st_mode=S_IFDIR|0755, st_size=4
0.000112 lstat("/mnt/var/www/html/drupal/sites/all/modules/Internet/date/date.module", {st_mode=S_IFREG|064
0.010816 open("/mnt/var/www/html/drupal/sites/all/modules/Internet/date/date.module", O_RDONLY) = 20
0.000835 fstat(20, {st_mode=S_IFREG|0644, st_size=44118, ...}) = 0
0.000069 lseek(20, 0, SEEK_CUR)    = 0
0.000063 stat("././sites/all/modules/Internet/date/date.module", {st_mode=S_IFREG|0644, st_size=44118, ...}) = 0
0.000159 close(20)                 = 0
...

As you can see, for every file included in the system there are around 10-15 lstat calls. this means that if we have ~200 files included we have ~2500 lstat calls per page. A simple sum on the strace results gave me ~8 seconds per page of lstat time, which was over a third of my test page time. see?

[user@host ~]# cat strace.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d |awk '{printf $2 "+"}'  | sed s/.$/\\n/ | bc
21.204960
[user@host ~]# cat strace.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d | fgrep lstat | awk '{printf $2 "+"}'  | sed s/.$/\\n/ | bc
8.361994

if you really want to understand this shell script let me know. It deserves a post by itself

Now we have 3 options: a) make lstat() faster. b) make php stop doing all those lstats(). c) make drupal include less files.

Well, (a) is not an option as our infrastructure team (aka sysadmins) insist there is nothing wrong with the filesystem and nothing can be done to enhance it's performance. Since I gave up being a sysadmin when it was still call 'sysadmin' it would be hard for me to prove them wrong (especially since they might be are right).

As (c) would require us to refactor most of our code in a very very 'not drupal way' and create very large php files, I was hoping to find a better solution.

By a rare chance of good fortune i came to learn about realpath_cache_size. Once I changed the foolish default to a reasonable 1M in php.ini, I lost most of the lstat() calls and over 10 seconds of processing time.

[user@host ~]# cat strace.2.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d |awk '{printf $2 "+"}'  | sed s/.$/\\n/ | bc
9.752309
[user@host ~]# cat strace.2.single | awk '{printf " " $1 "\n " $2}' |sed 1d | sed \$d | fgrep lstat | awk '{printf $2 "+"}'  | sed s/.$/\\n/ | bc
.040935

Here is what to do in php.ini

; ...php.ini...
; Determines the size of the realpath cache to be used by PHP. This value should
; be increased on systems where PHP opens many files to reflect the quantity of
; the file operations performed.
 realpath_cache_size=1M

; Duration of time, in seconds for which to cache realpath information for a given
; file or directory. For systems with rarely changing files, consider increasing this
; value.
realpath_cache_ttl=300

For normal pages under normal load I was able to get a 10%-25% performance improvment. I think that's quite a bit for a configuration change.

IE6 freeze with MOSS 2007

emposha wrote about how to handle a situtation where IE6 chokes on MOSS preview pages.

Hebrew SEO: גוגל, עברית, וה"א הידיעה

נראה ש-google לא מתייחס נכון לה"א הידיעה.
אם נחפש לדוגמא הממשלה נקבל בתוצאות את "פורטל השרותים של ממשלת ישראל" אבל גוגל מסמן בבולד רק את המילה הממשלה ולא את המילים ממשלה או ממשלת שהן הטיות חוקיות לחלוטין.
בכיוון ההפוך, אם נחפש ממשלה נקבל תוצאות אחרות לגמרי, כאשר המילים ממשלת וממשלה מודגשות אבל המילה הממשלה אינה מוגדשת.
לפי ההבדלים בתוצאות החיפוש נראה שהבעיה קיימת לא רק בהדגשה אלא גם באינדוקס.
בדקתי גם בחיפוש לפי הילדים מול ילדים ומילים נוספות. הדבר נראה נכון גם לגבי live של מיקרוסופט (בלי לדבר על yahooשבכלל לא טרחו לתמוך במורפולוגיה עברית)
המסקנה,לצרכי אופטימיזציה במנועי החיפוש של גוגל כדאי להגדיר את מילות המפתח של האתר עם ובלי ה"א הידיעה.