How to build Python libraries on 64bit Windows

The other day I needed to try some Python libraries at work. Being mostly .NET shop, we usually use Windows machines for development which sometimes causes problems for Linux-primary applications.
I'm running Windows 7 64bit and Python 2.7 64bit. Of course, the libraries were not built for this combination so I had to build them from source. Normally, this is a simple task of unzipping and running python setup.py install, but it was not that easy on this new machine.
After a lot of errors and pointless googling, I finally found a solution in an article Building Python Extensions in a Modern Windows Environment by Matt Schmidt. So if you have the same problem, you can just head there and read the article while I rant about the useless paths.

Fruitless paths

The problem manifested in a few different ways. Firstly, trying to build the libraries like usual threw error

raise ValueError(str(list(result.keys())))
ValueError: [u'path']

Not very useful message, mind you. There is a reported issue in Python for this exact exception during building libraries but it relates to different problem (using Visual Studio Express for building).
The problem appeared to be in Python's usage of Visual Studio compiler so I tried to use MinGW instead. What I did not verify was whether MinGW supports 64bit applications, or just 32bit. So instead of a built library I just had another error:

collect2: ld returned 1 exit status

I eventually got to the information that base MinGW can only be used for 32bit applications and even found a fork that should work for 64bits. I don't remember which fork it was, I only remember it not working. Fortunately I didn't have to fight with the libraries much longer because I finally stumbled on Matt's article.

Solution

This is a quick summary of Matt's article above:

1. Install Microsoft Visual C++ 2008 (sadly, VS 2010 is not supported at the moment)
   
2. Install the Windows 7 Platform SDK
   
3. Create directory C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\
   
4. Create file C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\vcvarsamd64.bat with following content:
   
call "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 /Release

   
5. Patch Python Visual studio compiler - add the following line to distutils/msvc9compiler.py after line 648 (this is in method link after line ld_args.append('/MANIFESTFILE:' + temp_manifest):
   

   
     ld_args.append('/MANIFEST')
     

   
   
6. That's it. Now you can run python setup.py install to install your brand new Python libraries


I hope this will be of use if you have similar problems, at least as a point to Matt's article.

Internet cookbook Edesia.cz

In the last post I mentioned that I started working on a new hobby project - internet cookbook Edesia.cz (in Czech language). I didn't write anything for some time but in the meantime successfully finished basic implementation with which I am quite satisfied - I strived for a clean UI with only important functionality and not much advertisements. But back to the beginning.

My original motivation to do this project had not much to do with cooking - I merely wanted to learn Python and Django. I realized that I don't want to fall into the trap of starting something new and never finishing (as was the case with Tomix). So this had to be a real project which I would have to be able to program in my spare time. I needed something relatively simple but with real value, I didn't want to develop yet another CMS or blog engine.

When I thought about the options I remembered that I already created a simple cookbook in C# a few years ago. I used it only for myself but the WinForms UI was very clunky and it had a few bugs I had no intention to fix. I decided I will just rewrite it for the web. I thought about basic functions I'd like to have in there, analyzed and draw wireframes.

The only problem I had to solve was how to motivate myself after the initial excitement fades out. I solved this by spending some time thinking about a reasonable name for this kind of website that was not yet taken and immediately bought the domain and paid for hosting (here "immediately" means about a month after I started development - I never said I was quick :) ). By the way, the site is named after Roman goddess of feasts and good food. So, I went on with the development. I didn't really bother with the UI design, knowing that I can't create anything nice and expected that I'll find some designer later. I found a reasonable free template (Styleshout's Keep It Simple) and by mid-April I deployed first simple version. That allowed me to fill in the recipes while working on other improvements. I did not really propagate the project so I was kinda excited when I saw that soon the visitors started coming on their own.

While working on important features, I was also looking for someone to design the final version of UI. I didn't want anything expensive because Edesia.cz is only a hobby project. Fortunately a friend of mine agreed to create the design and logo for a reasonable price. The work took longer than expected, partially because of my inexperience in this part of software development and partially because of some unexpected (work unrelated) issues. We did a few iterations of design, a few more of logo and during July site went public. Today the site is almost the same, only with a few bugs fixed and a few improvements implemented.

After the nice design went public, I registered the site to a few catalogues and promoted to friends on Facebook. I didn't do more marketing yet (mostly because I don't know how best to approach it) but so far the number of visitors to site is slowly increasing.

Today the site is still really small - I have over 350 recipes in multiple categories and about 80 visits per day. Two weeks ago I added some simple PPC advertisements because I'd like the site to at least pay for it's hosting costs. So far the results are not very exciting which is not surprising given the fact that the number of visits is still very low and the ads are at their first iteration. I don't want to have many advertisements there so I will probably have to bear the costs myself for some time.

Altogether, I am now satisfied with the results I saw so far. I am now able to do at least basic development in Django, I learned a little about SEO and specifics of web development and I also found quite a few good recipes to try :) For the future I plan to improve the rating system, add possibility to display recipe images, write articles and lots of other stuff. And by adding all this, I want the site to stay simple and clean unlike most of other recipe websites. I hope it'll work :)

Running Django on shared hosting with mod_wsgi

I have recently started working on a new hobby project (internet cookbook - only in Czech, nothing to see yet). This is written in Django and runs on shared webhosting (Web4ce). Django contains reasonable deployment documentation but unfortunately it deals only with configuring the site directly on Apache.

I can only modify .htaccess files so the deployment was a little harder than just copying a few lines of code. I had to actually google what to do :)

In the end the site is running, and this is what I did. After a few tries getting just 500 error (and not knowing where the problem was) I began with more simple steps then trying to set everything at once.

Simple WSGI script

First, to make sure that mod_wsgi is correctly supported and running, I created file hello_wsgi.wsgi:

def application(environ, start_response):
    status = '200 OK'
    output = 'Hello World!'

    response_headers = [('Content-type', 'text/plain'), ('Content-Length', str(len(output)))]
    start_response(status, response_headers)

    return [output]


and .htaccess:

AddHandler wsgi-script .wsgi


Uploading these files to hosting and navigating to hello_wsgi.wsgi gave me my desired "Hello World!". At least something was working.
The scripts for this part were taken from http://www.rkblog.rk.edu.pl/w/p/mod_wsgi/.

Running django

Preparing django.wsgi seems simple enough once the WSGI is actually working, but the problem is when there are some other problems in configuration (as it was in my case :)).
To be able to debug these problems, I have modified the suggested WSGI configuration:

import os
import sys

os.environ['DJANGO_SETTINGS_MODULE'] = 'edesia.settings'

sys.path.append('/path/to/my/www')

import django.core.handlers.wsgi
app = django.core.handlers.wsgi.WSGIHandler()

def application(environ, start_response):
    status = '200 OK'
        app(environ, start_response)
        output = 'ok'
    except Exception, e:
        output = 'error: %s ' % str(e)

    response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
    start_response(status, response_headers)

    return [output]


Running on django.wsgi like this, I get all error messages and can fix the problems as they appear. After I got 'ok' as output, I only need to rename "def application" to something like "application_debug" and "app" to "application". And, voilà, I am running Django powered website.

Serving static files with Apache

The only problem remaining was to configure Apache to serve static files directly and not through Django (it's possible, but slow - see Big Fat Disclaimer here: http://docs.djangoproject.com/en/1.1/howto/static-files/#howto-static-files ). Again, the original documentation was not very helpful, so I digged into Apache mod_rewrite and this is what I came up with:


RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(site_media/.*)$ /edesia/$1 [QSA,PT,L]


These two lines put into .htaccess cause all the static content (stored under directory site_media) to be served directly by apache from /edesia/site_media directory.

The whole .htaccess file then for me (and for now) looks like this:


DirectoryIndex django.wsgi

AddHandler wsgi-script .wsgi
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(site_media/.*)$ /edesia/$1 [QSA,PT,L]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ /django.wsgi/$1 [QSA,PT,L]


And that's it. The deployment took a little longer than I anticipated (plus I still have a lot more to do regarding deployment - see http://djangobook.com/en/2.0/chapter12/). But the site is running and that's fine with me now. Now I can concentrate on the development, find someone who will do the design and fill recipes. Hopefully by the beginning of spring I will have some reasonably complete version.

See also:
http://code.google.com/p/modwsgi/wiki/IntegrationWithDjango
http://code.google.com/p/modwsgi/wiki/ConfigurationGuidelines

Running xubuntu 9.10 from USB

I have recently bought a new USB flash disc to replace my old 512MB one. Having it, I naturally wanted to try booting from USB to use the new free space :) I decided to try "new" xubuntu 9.10 because I did not yet upgrade my desktop computer.

I have created new flash drive by the article on Pendrivelinux (Windows install seemed easier than burning LiveCD and creating USB from that). I tried to boot up. Unexpectedly, nothing happened. I guess the booting from USB is not yet as easy as using CDs. In the end I was successful and I want to sum up the problems I had and solutions to them.
For instructions how to create bootable USB disc, see www.pendrivelinux.com.

The solutions may be specific for motherboard Gigabyte EP45-DSR and other related boards.

Problem: booting from USB does not start
Solution: Don't try to set the boot order differently. That does not work (at least for my board). When booting starts, press F12 to go to boot menu. Here, choose "Hard Drive" and select your USB with the system.
Make sure that the USB is marked as USB-HDD0, otherwise the system won't boot from it. It seems that the motherboard checks just the first USB device it finds and ignores the rest.

Problem: booting starts, but after a while, message "end_request: I/O error, dev fd0, sector 0" is displayed periodically.
Solution: The Live USB is trying to find the floppy drive. Switch off the floppy drive in BIOS. In my case, this is done in menu Standard CMOS Features. Set option "Drive A" to "None".

I hope this helps if you have similar problems.