SMS are not 160 characters long, they are 140 bytes long! This is what I discovered today after my SO complained that her mobile operator was charging her for SMS she never sent…

And when you know how computers are working, it totally makes sense!

“So what?” are you going to ask? So, this is again a nice example of character encodings drive you crazy. According to wikipedia there are 3 encodings used in text messages which respectively use 7bits, 8bits and 16bits to encode a single character.

Depending on the characters you used in your message your phone is going to decide what encoding to use, thus reducing the maximum number of characters to, respectively, 160, 140 and 70 (and even less, see later). Any extra character will lead to the splitting of your message into multiple SMS and, obviously, a raise in your bill.

By default the 7bit encoding used is GSM 03.38, which has the following 128 characters alphabet: @, £, $, ¥, è, é, ù, ì, ò, Ç, LF, Ø, ø, CR, Å, å, Δ, _, Φ, Γ, Λ, Ω, Π, Ψ, Σ, Θ, Ξ, ESC, Æ, æ, ß, É, SP, !, “, #, ¤, %, &, ‘, (, ), *, +, ,, -, ., /, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, :, ;, <, =, >, ?, ¡, A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y, Z, Ä, Ö, Ñ, Ü, §, ¿, a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z, ä, ö, ñ, ü, à

If you use only those characters, then you text messages can have 160 characters, however, any character outside of this alphabet will mean the use of a different encoding. And if you are using exotic scripts, your messages will be encoded in UTF-16 and in this encoding a Chinese character, for example, will take up to 4 bytes, reducing the maximum length of you Chinese message to 35 characters max.

I guess that now that smart phones are supporting international scripts and transparently breaking up text messages, a lot of people get trapped. The only recommendation I can think of is to enable your phone to display the character count when you type text messages, I noticed that my iPhone is changing the maximum number of characters according to the encoding it’s going to use to send my message.

If you want to know more about character encodings I absolutely recommend the following article by Joel Spolsky: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

UPDATE: At Stephanie’s request here is how to activate message count on your iPhone (at least on my 3GS with iOS 4.1).

Go to your iPhone settings, scroll down to “Messages” then toggle “Character Count” on. When you write a text message the count will show up only if you have at least two lines of text :)

Related posts: (automatically generated)

• Jus d’Orange ou citron pressé?
• Fun with Java files encoding

It usually happen when the AppEngine team put the datastore in read-only mode for maintenance purpose but other capabilities can be tested as well.

def requires_datastore_write(view):
    def newview(request, *args, **kwargs):
        from google.appengine.api import capabilities
        datastore_write_enabled = capabilities.CapabilitySet('datastore_v3', capabilities=['write']).is_enabled()
 
        if datastore_write_enabled:
            return view(request, *args, **kwargs)
        else:
            from django.shortcuts import render_to_response
            from django.template import RequestContext
            return render_to_response('maintenance.html', context_instance=RequestContext(request))
 
    return newview

This is a python decorator and you can use it to decorate views that require write access to the datastore. For example:

@requires_datastore_write
def update(request):
    ...

You will need to create a Django template named
maintenance.html

to display a warning to your users. Mine looks like this:

Application Maintenance

The LibraryThing for Facebook application is currently
in maintenance mode and some operations are temporarily unavailable.
 
Thanks for trying back later. Sorry for the inconvenience.

Related posts: (automatically generated)

• A year in review
• Properly uploading files to Amazon S3
• I Can Haz Java?

It’s written in Python and does interesting things like compressing and minifying what needs to be. It takes 3 arguments and as 2 options:

Usage: s3uploader.py [-xm] src_folder destination_bucket_name prefix

src_folder

path to the local folder containing the static files to upload

destination_bucket_name

name of the S3 bucket to upload to (e.g. static.example.com)

prefix

a prefix to use for the destination key (kind of a folder on the destination bucket, I use it to specify a release version to defeat browser caching)

x

if set, the script will set a far future expiry for all files, otherwise the S3 default will be used (one day if I remember well)

m

if set, the script will minify css and javascript files

First you will have to install some dependencies, namely boto, jsmin and cssmin. Installation procedure will depend on your OS but on my Mac I do the following:

sudo easy_install boto
sudo easy_install jsmin
sudo easy_install cssmin

And here is the script itself:

#! /usr/bin/env python
import os, sys, boto, mimetypes, zipfile, gzip
from io import StringIO, BytesIO
from optparse import OptionParser
from jsmin import *
from cssmin import *
 
# Boto picks up configuration from the env.
os.environ['AWS_ACCESS_KEY_ID'] = 'Your AWS access key id goes here'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'Your AWS secret access key goes here'
 
# The list of content types to gzip, add more if needed
COMPRESSIBLE = [ 'text/plain', 'text/csv', 'application/xml',
                'application/javascript', 'text/css' ]
 
def main():
    parser = OptionParser(usage='usage: %prog [options] src_folder destination_bucket_name prefix')
    parser.add_option('-x', '--expires', action='store_true', help='set far future expiry for all files')
    parser.add_option('-m', '--minify', action='store_true', help='minify javascript files')
    (options, args) = parser.parse_args()
    if len(args) != 3:
        parser.error("incorrect number of arguments")
    src_folder = os.path.normpath(args[0])
    bucket_name = args[1]
    prefix = args[2]
 
    conn = boto.connect_s3()
    bucket = conn.get_bucket(bucket_name)
 
    namelist = []
    for root, dirs, files in os.walk(src_folder):
        if files and not '.svn' in root:
            path = os.path.relpath(root, src_folder)
            namelist += [os.path.normpath(os.path.join(path, f)) for f in files]
 
    print 'Uploading %d files to bucket %s' % (len(namelist), bucket.name)
    for name in namelist:
        content = open(os.path.join(src_folder, name))
        key = bucket.new_key(os.path.join(prefix, name))
        type, encoding = mimetypes.guess_type(name)
        type = type or 'application/octet-stream'
        headers = { 'Content-Type': type, 'x-amz-acl': 'public-read' }
        states = [type]
 
        if options.expires:
            # We only use HTTP 1.1 headers because they are relative to the time of download
            # instead of being hardcoded.
            headers['Cache-Control'] = 'max-age %d' % (3600 * 24 * 365)
 
        if options.minify and type == 'application/javascript':
            outs = StringIO()
            JavascriptMinify().minify(content, outs)
            content.close()
            content = outs.getvalue()
            if len(content) > 0 and content[0] == '\n':
                content = content[1:]
            content = BytesIO(content)
            states.append('minified')
 
        if options.minify and type == 'text/css':
            outs = cssmin(content.read())
            content.close()
            content = outs
            if len(content) > 0 and content[0] == '\n':
                content = content[1:]
            content = BytesIO(content)
            states.append('minified')
 
        if type in COMPRESSIBLE:
            headers['Content-Encoding'] = 'gzip'
            compressed = StringIO()
            gz = gzip.GzipFile(filename=name, fileobj=compressed, mode='w')
            gz.writelines(content)
            gz.close()
            content.close
            content = BytesIO(compressed.getvalue())
            states.append('gzipped')
 
        states = ', '.join(states)
        print '- %s => %s (%s)' % (name, key.name, states)
        key.set_contents_from_file(content, headers)
        content.close();
 
if __name__ == '__main__':
    main()

Thanks to Nico for the expiry trick :)

Related posts: (automatically generated)

• A year in review
• How to manage Google AppEngine maintenance periods

There has been a lot of comments already about Sidewiki but the thing that instantly stroke me is the fact that there’s no easy way to keep up with what others are saying about your own pages. So I took a look at the Sidewiki API and built the Sidewiki RSS service.

This free service (hope you won’t mind the Google Ads) enables webmasters to get the URL to the recent Sidewiki entries for their pages. There’s even a bookmarklet that you can drop in your browser’s toolbar and use to get the feed of the page you are browsing.

Hope you will like it ;)

Related posts: (automatically generated)

• Spare me the talk about privacy, they’re all clueless anyway…
• A year in review
• Does Google read this blog?

I chose Apache JMeter to generate the load, created a few test plans and started to nudge the application. Of course I quickly understood that my MacBook won’t be enough have the server sweat.

To serve this application we are using Amazon EC2 instances managed with the Sclar.net service. One day I should write something about Scalr, but for now, enough is to say that a Scalr farm defines the configuration of a cluster of EC2 instances. Each instance in a farm belongs to a role (an EC2 AMI) and the farm defines how many instances of each role should be launched.

Since JMeter can be used in a master/slave mode (see details here) I decided to use Scalr to create a farm of JMeter slaves that would put the application under pressure.

The first problem I faced is that the JMeter master and its slaves must be in the same sub-network to be able to communicate, so my JMeter farm had to define two different roles, one for the master (jmeter-master) with only one instance and one for the slaves (jmeter-slave) with as many instances as necessary.

The second problem was concerning the IP addresses of the slaves, I did not want to write down the slaves’ IPs and manually enter them in the JMeter command line. By chance, with Scalr, each instance in a farm is informed of its peers’ IP addresses, so I wrote a small Python script that would get those IPs and launch the JMeter master with a given test plan.

#! /usr/bin/python
import os, sys, subprocess, datetime
 
JMETER_CMD = '/usr/share/jmeter/bin/jmeter'
SCRIPTS_ROOT = '/var/testing/'
# Instance IPs for a given role are filenames in the '/etc/aws/hosts' folder
SLAVES = os.listdir('/etc/aws/hosts/jmeter-slave')
 
def jmeter(script):
    logname = datetime.datetime.now().strftime('%Y%m%d%H%M%S') + '.log'
    script = os.path.join(SCRIPTS_ROOT, script)
    cmd = [ JMETER_CMD, '-n' ]
    cmd += [ '-t', script ]
    cmd += [ '-R', ','.join(SLAVES) ]
    cwd = SCRIPTS_ROOT
    subprocess.check_call(cmd, cwd=cwd, stderr=sys.stderr, stdout=sys.stdout)
 
if __name__ == '__main__':
    jmeter(sys.argv[1])

This was working pretty nicely for my simpler test plans (like the one that only GETs the home page) but as soon as I tried to POST (like during the login process) this was not enough. The thing is that the POST data that JMeter is using are not stored in the test plan itself but in companion .binary files, and those files are not sent by the master to the slaves like the test plans are.

I thus had to find a way to send those files by myself before the launch of the test plans. Rsync seemed the easiest thing to do, so I wrote another Python script to synchronize the slaves.

#! /usr/bin/python
import os, sys, subprocess
 
SCRIPTS_ROOT = '/var/testing/'
# Instance IPs for a given role are filenames in the '/etc/aws/hosts' folder
SLAVES = os.listdir('/etc/aws/hosts/jmeter-slave')
 
def sync():
    for slave in SLAVES:
        dest = '%s:/var/testing' % slave
        cmd = ( 'rsync', '-r', '-e', 'ssh -q -i /var/testing/farm.key', SCRIPT_ROOT, dest)
        subprocess.check_call(cmd, stderr=sys.stderr, stdout=sys.stdout)
 
if __name__ == '__main__':
    sync()

The above script requires only three things:

• a valid RSA private key (here /var/testing/farm.key), which you can download using the Scalr.net farm’s menu
• the /var/testing folder must already exist on the slaves
• and, of course, you need to initially get the files on the master. I use svn up.

Once you have prepared and tested everything, using one master and one slave, you can rebundle the instances you used and then start to spawn tens of slaves to stress your application.

If you have already done something similar or have ideas for improving my setup do not hesitate in letting me know in the comments :)

UPDATE: With the release of the Amazon Virtual Private Cloud it should be possible now to have slaves running in the cloud and a master running on your workstation, they would all be in your own sub-network. However, you will need to find another way to synchronize the POST data with the slaves.

Most Commented Posts

• Sévices Après Vente
• JMeter distributed testing with Amazon EC2
• Google, please, NO!

The following code is used to post two different fields, a simple text data named “somedata” and a file named “somefile”.
Hope it helps :)

$destination = "http://yourdomain.com/yoururl";
 
$eol = "\r\n";
$data = '';
 
$mime_boundary=md5(time());
 
$data .= '--' . $mime_boundary . $eol;
$data .= 'Content-Disposition: form-data; name="somedata"' . $eol . $eol;
$data .= "Some Data" . $eol;
$data .= '--' . $mime_boundary . $eol;
$data .= 'Content-Disposition: form-data; name="somefile"; filename="filename.ext"' . $eol;
$data .= 'Content-Type: text/plain' . $eol;
$data .= 'Content-Transfer-Encoding: base64' . $eol . $eol;
$data .= chunk_split(base64_encode("Some file content")) . $eol;
$data .= "--" . $mime_boundary . "--" . $eol . $eol; // finish with two eol's!!
 
$params = array('http' => array(
                  'method' => 'POST',
                  'header' => 'Content-Type: multipart/form-data; boundary=' . $mime_boundary . $eol,
                  'content' => $data
               ));
 
$ctx = stream_context_create($params);
$response = @file_get_contents($destination, FILE_TEXT, $ctx);

Most Commented Posts

• Sévices Après Vente
• JMeter distributed testing with Amazon EC2
• Google, please, NO!

It comes with a set of Eclipse plugins to test and deploy Java servlets, using JDO or JPA to support database access. Of course, the database behind this is BigTable, which means that a lot of relational features are not available, but it scales!

Go there to get you started, or, if you want to know if your preferred framework will play well with GAE, go to the “Will it play in App Engine” page.

That’s good news! Especially because we may start having more and more Java applications outside of the corporate walls.

Yes Google, YES!

Related posts: (automatically generated)

• How to manage Google AppEngine maintenance periods
• Spare me the talk about privacy, they’re all clueless anyway…
• Sidewiki RSS

However, I wanted a deeper integration between this blog and my Facebook profile. Here is what I have done.

In the process of installing the fbConnect plugin you have to create a Facebook application and, among other things, Facebook applications feature a canvas page and an application tab. The canvas page is the main application page, the tab can be added to any user’s or page’s profiles.

To setup the canvas page you specify an URL that will either serve FBML (Facebook markup) or pure HTML. In the latter case the page is displayed in an IFRAME.

Initially, I used the IFRAME version to display the home page but I found awkward to have my blog design mixed with the Facebook design. Additionally this technique cannot be used for tabs, that only support FBML.

Finally, I crafted special pages on this blog that serves only FBML with a special Facebook styling extracted from Foxinni’s Facebook WordPress Theme. The resulting canvas page can bee seen here and the corresponding profile tab is now featured on my own Facebook profile, here (you may not be able to see this one because of Facebook privacy control so I inserted a screenshot below).

If this is getting enough interest I might package it as a WordPress plugin. Leave a comment if you are interested or if you have additional ideas.

UPDATE (2010-03-13): This is now a WordPress plugin, see my plugins page.

Related posts: (automatically generated)

• Facebook connect for Wordpress
• Facebook Pages Notifications
• Experiencing viral growth

For those who do not know what the SETI@Home project is, the goal is to sort out radio signals from the Arecibo radio telescope using grid computing in order to find extraterrestrial signals.

As far as I know this was the very first large scale implementation of a grid based computer and today, with 556,888 machines and 1,393.74 TeraFLOPS per day, I think this can be considered the most powerful super-computer in the world.

We still did not find any track of Samantha Mulder, but what this project has actually proven is that it is possible to use desktop computers idle time to do useful scientific researches.

In 2005, the SETI@Home software became BOINC (Berkeley Open Infrastructure for Network Computing) and mutated into a platform which is now not only supporting the SETI@Home project but many other scientific projects.

When I am not using them, my laptop and my desktop are running the following projects:

• SETI@Home: detection of extraterrestrial technology.
• climateprediction.net: the largest experiment to produce a forecast of the climate in the 21st century.
• Einstein@Home:  gravitational waves detection.
• LHC@Home: processing LHC data to detect Higgs boson and other funny particles.

What are you doing with your computers’ idle time? If you have BOINC installed, list the projects you are supporting in the comments, if not then install it and become part of the most powerful super-computer in the Solar System ;)

Most Commented Posts

• Sévices Après Vente
• JMeter distributed testing with Amazon EC2
• Google, please, NO!

To be able to code and unit test you part you have to make assumptions about the workings of this system, deploy in a real environment (which can be quite time consuming) and pray.

This is like trying to shoot a bear in the dark, if you miss it the first time it will run after you like a delivery deadline. The only thing you can do is try again until you hit or it passes you by…

If you feel any relief in sharing your dreadiest experiences feel free to comment :)

Related posts: (automatically generated)

• XML schemas compatibility
• The Human ESB
• Database schemas compatibility