How to build Python libraries on 64bit Windows

The other day I needed to try some Python libraries at work. Being mostly .NET shop, we usually use Windows machines for development which sometimes causes problems for Linux-primary applications.
I'm running Windows 7 64bit and Python 2.7 64bit. Of course, the libraries were not built for this combination so I had to build them from source. Normally, this is a simple task of unzipping and running python setup.py install, but it was not that easy on this new machine.
After a lot of errors and pointless googling, I finally found a solution in an article Building Python Extensions in a Modern Windows Environment by Matt Schmidt. So if you have the same problem, you can just head there and read the article while I rant about the useless paths.

Fruitless paths

The problem manifested in a few different ways. Firstly, trying to build the libraries like usual threw error

raise ValueError(str(list(result.keys())))
ValueError: [u'path']

Not very useful message, mind you. There is a reported issue in Python for this exact exception during building libraries but it relates to different problem (using Visual Studio Express for building).
The problem appeared to be in Python's usage of Visual Studio compiler so I tried to use MinGW instead. What I did not verify was whether MinGW supports 64bit applications, or just 32bit. So instead of a built library I just had another error:

collect2: ld returned 1 exit status

I eventually got to the information that base MinGW can only be used for 32bit applications and even found a fork that should work for 64bits. I don't remember which fork it was, I only remember it not working. Fortunately I didn't have to fight with the libraries much longer because I finally stumbled on Matt's article.

Solution

This is a quick summary of Matt's article above:

1. Install Microsoft Visual C++ 2008 (sadly, VS 2010 is not supported at the moment)
   
2. Install the Windows 7 Platform SDK
   
3. Create directory C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\
   
4. Create file C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\vcvarsamd64.bat with following content:
   
call "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 /Release

   
5. Patch Python Visual studio compiler - add the following line to distutils/msvc9compiler.py after line 648 (this is in method link after line ld_args.append('/MANIFESTFILE:' + temp_manifest):
   

   
     ld_args.append('/MANIFEST')
     

   
   
6. That's it. Now you can run python setup.py install to install your brand new Python libraries


I hope this will be of use if you have similar problems, at least as a point to Matt's article.

Internet cookbook Edesia.cz

In the last post I mentioned that I started working on a new hobby project - internet cookbook Edesia.cz (in Czech language). I didn't write anything for some time but in the meantime successfully finished basic implementation with which I am quite satisfied - I strived for a clean UI with only important functionality and not much advertisements. But back to the beginning.

My original motivation to do this project had not much to do with cooking - I merely wanted to learn Python and Django. I realized that I don't want to fall into the trap of starting something new and never finishing (as was the case with Tomix). So this had to be a real project which I would have to be able to program in my spare time. I needed something relatively simple but with real value, I didn't want to develop yet another CMS or blog engine.

When I thought about the options I remembered that I already created a simple cookbook in C# a few years ago. I used it only for myself but the WinForms UI was very clunky and it had a few bugs I had no intention to fix. I decided I will just rewrite it for the web. I thought about basic functions I'd like to have in there, analyzed and draw wireframes.

The only problem I had to solve was how to motivate myself after the initial excitement fades out. I solved this by spending some time thinking about a reasonable name for this kind of website that was not yet taken and immediately bought the domain and paid for hosting (here "immediately" means about a month after I started development - I never said I was quick :) ). By the way, the site is named after Roman goddess of feasts and good food. So, I went on with the development. I didn't really bother with the UI design, knowing that I can't create anything nice and expected that I'll find some designer later. I found a reasonable free template (Styleshout's Keep It Simple) and by mid-April I deployed first simple version. That allowed me to fill in the recipes while working on other improvements. I did not really propagate the project so I was kinda excited when I saw that soon the visitors started coming on their own.

While working on important features, I was also looking for someone to design the final version of UI. I didn't want anything expensive because Edesia.cz is only a hobby project. Fortunately a friend of mine agreed to create the design and logo for a reasonable price. The work took longer than expected, partially because of my inexperience in this part of software development and partially because of some unexpected (work unrelated) issues. We did a few iterations of design, a few more of logo and during July site went public. Today the site is almost the same, only with a few bugs fixed and a few improvements implemented.

After the nice design went public, I registered the site to a few catalogues and promoted to friends on Facebook. I didn't do more marketing yet (mostly because I don't know how best to approach it) but so far the number of visitors to site is slowly increasing.

Today the site is still really small - I have over 350 recipes in multiple categories and about 80 visits per day. Two weeks ago I added some simple PPC advertisements because I'd like the site to at least pay for it's hosting costs. So far the results are not very exciting which is not surprising given the fact that the number of visits is still very low and the ads are at their first iteration. I don't want to have many advertisements there so I will probably have to bear the costs myself for some time.

Altogether, I am now satisfied with the results I saw so far. I am now able to do at least basic development in Django, I learned a little about SEO and specifics of web development and I also found quite a few good recipes to try :) For the future I plan to improve the rating system, add possibility to display recipe images, write articles and lots of other stuff. And by adding all this, I want the site to stay simple and clean unlike most of other recipe websites. I hope it'll work :)

Running Django on shared hosting with mod_wsgi

I have recently started working on a new hobby project (internet cookbook - only in Czech, nothing to see yet). This is written in Django and runs on shared webhosting (Web4ce). Django contains reasonable deployment documentation but unfortunately it deals only with configuring the site directly on Apache.

I can only modify .htaccess files so the deployment was a little harder than just copying a few lines of code. I had to actually google what to do :)

In the end the site is running, and this is what I did. After a few tries getting just 500 error (and not knowing where the problem was) I began with more simple steps then trying to set everything at once.

Simple WSGI script

First, to make sure that mod_wsgi is correctly supported and running, I created file hello_wsgi.wsgi:

def application(environ, start_response):
    status = '200 OK'
    output = 'Hello World!'

    response_headers = [('Content-type', 'text/plain'), ('Content-Length', str(len(output)))]
    start_response(status, response_headers)

    return [output]


and .htaccess:

AddHandler wsgi-script .wsgi


Uploading these files to hosting and navigating to hello_wsgi.wsgi gave me my desired "Hello World!". At least something was working.
The scripts for this part were taken from http://www.rkblog.rk.edu.pl/w/p/mod_wsgi/.

Running django

Preparing django.wsgi seems simple enough once the WSGI is actually working, but the problem is when there are some other problems in configuration (as it was in my case :)).
To be able to debug these problems, I have modified the suggested WSGI configuration:

import os
import sys

os.environ['DJANGO_SETTINGS_MODULE'] = 'edesia.settings'

sys.path.append('/path/to/my/www')

import django.core.handlers.wsgi
app = django.core.handlers.wsgi.WSGIHandler()

def application(environ, start_response):
    status = '200 OK'
        app(environ, start_response)
        output = 'ok'
    except Exception, e:
        output = 'error: %s ' % str(e)

    response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
    start_response(status, response_headers)

    return [output]


Running on django.wsgi like this, I get all error messages and can fix the problems as they appear. After I got 'ok' as output, I only need to rename "def application" to something like "application_debug" and "app" to "application". And, voilà, I am running Django powered website.

Serving static files with Apache

The only problem remaining was to configure Apache to serve static files directly and not through Django (it's possible, but slow - see Big Fat Disclaimer here: http://docs.djangoproject.com/en/1.1/howto/static-files/#howto-static-files ). Again, the original documentation was not very helpful, so I digged into Apache mod_rewrite and this is what I came up with:


RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(site_media/.*)$ /edesia/$1 [QSA,PT,L]


These two lines put into .htaccess cause all the static content (stored under directory site_media) to be served directly by apache from /edesia/site_media directory.

The whole .htaccess file then for me (and for now) looks like this:


DirectoryIndex django.wsgi

AddHandler wsgi-script .wsgi
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(site_media/.*)$ /edesia/$1 [QSA,PT,L]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ /django.wsgi/$1 [QSA,PT,L]


And that's it. The deployment took a little longer than I anticipated (plus I still have a lot more to do regarding deployment - see http://djangobook.com/en/2.0/chapter12/). But the site is running and that's fine with me now. Now I can concentrate on the development, find someone who will do the design and fill recipes. Hopefully by the beginning of spring I will have some reasonably complete version.

See also:
http://code.google.com/p/modwsgi/wiki/IntegrationWithDjango
http://code.google.com/p/modwsgi/wiki/ConfigurationGuidelines

Running xubuntu 9.10 from USB

I have recently bought a new USB flash disc to replace my old 512MB one. Having it, I naturally wanted to try booting from USB to use the new free space :) I decided to try "new" xubuntu 9.10 because I did not yet upgrade my desktop computer.

I have created new flash drive by the article on Pendrivelinux (Windows install seemed easier than burning LiveCD and creating USB from that). I tried to boot up. Unexpectedly, nothing happened. I guess the booting from USB is not yet as easy as using CDs. In the end I was successful and I want to sum up the problems I had and solutions to them.
For instructions how to create bootable USB disc, see www.pendrivelinux.com.

The solutions may be specific for motherboard Gigabyte EP45-DSR and other related boards.

Problem: booting from USB does not start
Solution: Don't try to set the boot order differently. That does not work (at least for my board). When booting starts, press F12 to go to boot menu. Here, choose "Hard Drive" and select your USB with the system.
Make sure that the USB is marked as USB-HDD0, otherwise the system won't boot from it. It seems that the motherboard checks just the first USB device it finds and ignores the rest.

Problem: booting starts, but after a while, message "end_request: I/O error, dev fd0, sector 0" is displayed periodically.
Solution: The Live USB is trying to find the floppy drive. Switch off the floppy drive in BIOS. In my case, this is done in menu Standard CMOS Features. Set option "Drive A" to "None".

I hope this helps if you have similar problems.

Simple ICQ Export script

Recently my girlfriend finally decided to stop using the original ICQ program, which she retained mostly because of the unimaginable loss of conversation history. Quick search did not reveal any good and simple means to convert the history data. Most of the software was outdated or for different versions.

We had two versions of history files - one was directory with XML files and the other seems to be Access mdb (created by ICQ version 6 I think). To sharpen my Python skills (which at the time being is a bit like trying to sharpen a club), I decided to write simple conversion script for the XML history files.

The script is published on http://code.google.com/p/icqexport/. For convenience, I have also prepared Windows build http://icqexport.googlecode.com/files/icqexport_01.zip. You are free to do anything with the script and welcome to send me notes and suggestions.

The usage is very simple, although nothing is guaranteed :) Copy the script to the directory where ICQ history files are stored and run it. It will traverse all subdirectories and convert history XML files to one readable TXT file.

The only version supported is 0x00140034 (tested on ICQ version 5.1). You're fine if there's something like this at the beginning of your files:

<root>
<version>0x00140034</version>
<event>
...

I didn't see the mdb structure yet but if I will get to it, I'll keep you informed.

Tomix - First level ready

During this weekend I planned to finish the first version of first level of our puzzle game. I had kinda slacking mood (caused partially by this great blog), but I forced myself to overcome it and in the end I was able to finish the first level as planned. There are still no graphics so I'm reusing arrows from Heartbeat, but the basic "engine" is ready. There is still a lot of work to do but being able to actually play the level is really motivational.

I handed it over to Mitsuki to try the game logic and she quickly found a solution - although not the one I had originally envisioned :) But it was possible because of a bug in checking of the final position so in the next version it won't count. But at first glance it seems that the idea is viable and I have a lot of great feedback for the game from Mitsuki. We only have to stay focused on the fact that we create puzzle game, because some ideas would lead us to completely different genre :)

So now Mitsuki will start working on the graphics to incorporate to the game. In the evening I also checked some sites with free game development resources so we could re-use something already created, but my search did not finish yet. But I have a few good sources I want to check so expect another post covering those I think are most useful.

For the next work I hope to finally start work on the editor. I kinda didn't get to that yet, but I already know how to serialize objects in Python :) And it has not that high priority for me - worst case we can easily define the levels in source code.

I also checked some libraries to create menus in pygame but unfortunately none of them seem appropriate to what I want to do so I'll probably have to write my own solution. KezMenu looks fine but it can't center the text and unfortunately it's GPL'd - and I did not decide on the license to code yet, so I may not be able to use that. Well, we'll see.

Device identification problem during boot

Since installing xubuntu last autumn, it sometimes happened to me that my machine required several restarts to boot up. I have identified the problem but being linux n00b I was unable to find some solution to it for some time.

The problem was following:
I have two harddiscs in my computer - one is old IDE disc and the other is new SATA II disc. The Linux system is installed on the new disc which is usually identified as /dev/sda (where /dev/sda5 is the root partition). The other disc is usually on /dev/sdb. What sometimes happened was that the devices where identified the other way round - IDE being /dev/sda and SATA being /dev/sdb. After that the computer of course did not boot and ended in some sort of fallback console (sorry, I forgot the mode's name).

Sadly, asking administrators at work did not help me (most probably because I was not able to describe the problem correctly - the only thing I found out was that the IDE disc should be /dev/hd something and not /dev/sd something - very helpful).

Before some time I found what I thought will lead to solution, but I was hesitant to change the fstab because I didn't want to break it any more than it was :) What I needed was to identify the mount points by partition label instead of device (see http://tldp.org/HOWTO/html_single/Partition/#labels).

Today, I finally got to fixing it (at least, it appears to be fixed now :)). The whole process involved:

• installing GParted which unfortunately did not help me with setting the partition labels and thus was not that useful (it provided me with nice overview of the partitions :))
• setting label to ext3 partitions using tune2fs (e.g. tune2fs -L pubsw /dev/hdb1 sets label pubsw to par
• setting labels to NTFS partitions using ntfslabel (part of ntfsprogs package)
• modifying fstab to use the labels (see http://tldp.org/HOWTO/html_single/Partition/#labels for example fstab)

To sum it up - if you have this problem with identifying devices during boot up, use labels to identify them in fstab. This seems like a good idea generally because as I understand it the device identification can change when switching hardware (imagine adding shiny new HDD to your computer and not being able to boot up as the result - ouch!)

In the end the changes where really not that hard and if I knew a little about Linux I could've fixed this long time ago. But I'm a little smarter now and maybe after some time I will know Linux as good as Windows. Most probably after Windows 7 comes out - because last version I had is Windows XP, I will know very little about both Windows and Linux.