These days we should write every code as if it will be used by international people with a wide variety of personal information (just look at Falsehoods Programmers Believe About Names for some headscratchers). I would like to do add my small contribution to this by showing how UTF-8 encoded strings can be passed into GET/POST parameters.

For this I’ll be using the following small PHP script, which can be quickly run by the command line PHP webserver added in PHP 5.4:

GETs: 
POSTs:

We’ll test this with the following Python script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
#!/usr/bin/python
# vim: set fileencoding=utf-8 :
import urllib
import urllib2

params = {'name': u'東京'}
params = { k: v.encode('utf-8') for k, v in params.iteritems() }
data = urllib.urlencode(params)

url = 'http://localhost:8000/?' + data
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)

print response.read()

This all works well and nicely, so here are some conclusions:

  • GET and POST variables need to be UTF-8 encoded after which they need to be urlencoded (“% encoded”). See this StackOverflow answer.
  • Based on the same answer: hostnames use Punycode instead (but we are not concerned with hostnames here)
  • You might need to add the following header for POST requests to work: “Content-Type: application/x-www-form-urlencoded; charset=UTF-8”
  • Failing to observe this sequence leads to an UnicodeEncodeError in urllib.urlencode