Aredhel's Red Hell

How to build Python libraries on 64bit Windows

2010-12-08T22:45:00.005+01:00

The other day I needed to try some Python libraries at work. Being mostly .NET shop, we usually use Windows machines for development which sometimes causes problems for Linux-primary applications.
I'm running Windows 7 64bit and Python 2.7 64bit. Of course, the libraries were not built for this combination so I had to build them from source. Normally, this is a simple task of unzipping and running python setup.py install, but it was not that easy on this new machine.
After a lot of errors and pointless googling, I finally found a solution in an article Building Python Extensions in a Modern Windows Environment by Matt Schmidt. So if you have the same problem, you can just head there and read the article while I rant about the useless paths.

Fruitless paths

The problem manifested in a few different ways. Firstly, trying to build the libraries like usual threw error


raise ValueError(str(list(result.keys())))
ValueError: [u'path']

Not very useful message, mind you. There is a reported issue in Python for this exact exception during building libraries but it relates to different problem (using Visual Studio Express for building).
The problem appeared to be in Python's usage of Visual Studio compiler so I tried to use MinGW instead. What I did not verify was whether MinGW supports 64bit applications, or just 32bit. So instead of a built library I just had another error:


collect2: ld returned 1 exit status

I eventually got to the information that base MinGW can only be used for 32bit applications and even found a fork that should work for 64bits. I don't remember which fork it was, I only remember it not working. Fortunately I didn't have to fight with the libraries much longer because I finally stumbled on Matt's article.

Solution

This is a quick summary of Matt's article above:

Install Microsoft Visual C++ 2008 (sadly, VS 2010 is not supported at the moment)
Install the Windows 7 Platform SDK
Create directory C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\
Create file C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\vcvarsamd64.bat with following content:
call "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 /Release
Patch Python Visual studio compiler - add the following line to distutils/msvc9compiler.py after line 648 (this is in method link after line ld_args.append('/MANIFESTFILE:' + temp_manifest): ld_args.append('/MANIFEST')
That's it. Now you can run python setup.py install to install your brand new Python libraries



I hope this will be of use if you have similar problems, at least as a point to Matt's article.



Internet cookbook Edesia.cz
2010-10-28T12:33:00.003+02:00
In the last post I mentioned that I started working on a new hobby project - internet cookbook Edesia.cz (in Czech language). I didn't write anything for some time but in the meantime successfully finished basic implementation with which I am quite satisfied - I strived for a clean UI with only important functionality and not much advertisements. But back to the beginning.

My original motivation to do this project had not much to do with cooking - I merely wanted to learn Python and Django. I realized that I don't want to fall into the trap of starting something new and never finishing (as was the case with Tomix). So this had to be a real project which I would have to be able to program in my spare time. I needed something relatively simple but with real value, I didn't want to develop yet another CMS or blog engine.

When I thought about the options I remembered that I already created a simple cookbook in C# a few years ago. I used it only for myself but the WinForms UI was very clunky and it had a few bugs I had no intention to fix. I decided I will just rewrite it for the web. I thought about basic functions I'd like to have in there, analyzed and draw wireframes. 

The only problem I had to solve was how to motivate myself after the initial excitement fades out. I solved this by spending some time thinking about a reasonable name for this kind of website that was not yet taken and immediately bought the domain and paid for hosting (here "immediately" means about a month after I started development - I never said I was quick :) ). By the way, the site is named after Roman goddess of feasts and good food. So, I went on with the development. I didn't really bother with the UI design, knowing that I can't create anything nice and expected that I'll find some designer later. I found a reasonable free template (Styleshout's Keep It Simple) and by mid-April I deployed first simple version. That allowed me to fill in the recipes while working on other improvements. I did not really propagate the project so I was kinda excited when I saw that soon the visitors started coming on their own.

While working on important features, I was also looking for someone to design the final version of UI. I didn't want anything expensive because Edesia.cz is only a hobby project. Fortunately a friend of mine agreed to create the design and logo for a reasonable price. The work took longer than expected, partially because of my inexperience in this part of software development and partially because of some unexpected (work unrelated) issues. We did a few iterations of design, a few more of logo and during July site went public. Today the site is almost the same, only with a few bugs fixed and a few improvements implemented.

After the nice design went public, I registered the site to a few catalogues and promoted to friends on Facebook. I didn't do more marketing yet (mostly because I don't know how best to approach it) but so far the number of visitors to site is slowly increasing.

Today the site is still really small - I have over 350 recipes in multiple categories and about 80 visits per day. Two weeks ago I added some simple PPC advertisements because I'd like the site to at least pay for it's hosting costs. So far the results are not very exciting which is not surprising given the fact that the number of visits is still very low and the ads are at their first iteration. I don't want to have many advertisements there so I will probably have to bear the costs myself for some time.

Altogether, I am now satisfied with the results I saw so far. I am now able to do at least basic development in Django, I learned a little about SEO and specifics of web development and I also found quite a few good recipes to try :) For the future I plan to improve the rating system, add possibility to display recipe images, write articles and lots of other stuff. And by adding all this, I want the site to stay simple and clean unlike most of other recipe websites. I hope it'll work :)


Running Django on shared hosting with mod_wsgi
2010-02-09T10:26:00.010+01:00
I have recently started working on a new hobby project (internet cookbook - only in Czech, nothing to see yet). This is written in Django and runs on shared webhosting (Web4ce). Django contains reasonable deployment documentation but unfortunately it deals only with configuring the site directly on Apache.

I can only modify .htaccess files so the deployment was a little harder than just copying a few lines of code. I had to actually google what to do :)

In the end the site is running, and this is what I did. After a few tries getting just 500 error (and not knowing where the problem was) I began with more simple steps then trying to set everything at once.

Simple WSGI script

First, to make sure that mod_wsgi is correctly supported and running, I created file hello_wsgi.wsgi:

def application(environ, start_response):
    status = '200 OK'
    output = 'Hello World!'

    response_headers = [('Content-type', 'text/plain'), ('Content-Length', str(len(output)))]
    start_response(status, response_headers)

    return [output]


and .htaccess:

AddHandler wsgi-script .wsgi


Uploading these files to hosting and navigating to hello_wsgi.wsgi gave me my desired "Hello World!". At least something was working.
The scripts for this part were taken from http://www.rkblog.rk.edu.pl/w/p/mod_wsgi/.

Running django

Preparing django.wsgi seems simple enough once the WSGI is actually working, but the problem is when there are some other problems in configuration (as it was in my case :)).
To be able to debug these problems, I have modified the suggested WSGI configuration:

import os
import sys

os.environ['DJANGO_SETTINGS_MODULE'] = 'edesia.settings'

sys.path.append('/path/to/my/www')

import django.core.handlers.wsgi
app = django.core.handlers.wsgi.WSGIHandler()

def application(environ, start_response):
    status = '200 OK'
        app(environ, start_response)
        output = 'ok'
    except Exception, e:
        output = 'error: %s ' % str(e)

    response_headers = [('Content-type', 'text/plain'),
                      ('Content-Length', str(len(output)))]
    start_response(status, response_headers)

    return [output]


Running on django.wsgi like this, I get all error messages and can fix the problems as they appear. After I got 'ok' as output, I only need to rename "def application" to something like "application_debug" and "app" to "application". And, voilà, I am running Django powered website.

Serving static files with Apache

The only problem remaining was to configure Apache to serve static files directly and not through Django (it's possible, but slow - see Big Fat Disclaimer here: http://docs.djangoproject.com/en/1.1/howto/static-files/#howto-static-files ). Again, the original documentation was not very helpful, so I digged into Apache mod_rewrite and this is what I came up with:


RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(site_media/.*)$ /edesia/$1 [QSA,PT,L]


These two lines put into .htaccess cause all the static content (stored under directory site_media) to be served directly by apache from /edesia/site_media directory.

The whole .htaccess file then for me (and for now) looks like this:


DirectoryIndex django.wsgi

AddHandler wsgi-script .wsgi
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(site_media/.*)$ /edesia/$1 [QSA,PT,L]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ /django.wsgi/$1 [QSA,PT,L]


And that's it. The deployment took a little longer than I anticipated (plus I still have a lot more to do regarding deployment - see http://djangobook.com/en/2.0/chapter12/). But the site is running and that's fine with me now. Now I can concentrate on the development, find someone who will do the design and fill recipes. Hopefully by the beginning of spring I will have some reasonably complete version.

See also:
http://code.google.com/p/modwsgi/wiki/IntegrationWithDjango
http://code.google.com/p/modwsgi/wiki/ConfigurationGuidelines


Running xubuntu 9.10 from USB
2010-01-21T17:05:00.008+01:00
I have recently bought a new USB flash disc to replace my old 512MB one. Having it, I naturally wanted to try booting from USB to use the new free space :) I decided to try "new" xubuntu 9.10 because I did not yet upgrade my desktop computer.

I have created new flash drive by the article on Pendrivelinux  (Windows install seemed easier than burning LiveCD and creating USB from that). I tried to boot up. Unexpectedly, nothing happened. I guess the booting from USB is not yet as easy as using CDs. In the end I was successful and I want to sum up the problems I had and solutions to them.
For instructions how to create bootable USB disc, see www.pendrivelinux.com.

The solutions may be specific for motherboard Gigabyte EP45-DSR and other related boards.

Problem: booting from USB does not start
Solution: Don't try to set the boot order differently. That does not work (at least for my board). When booting starts, press F12 to go to boot menu. Here, choose "Hard Drive" and select your USB with the system.
Make sure that the USB is marked as USB-HDD0, otherwise the system won't boot from it. It seems that the motherboard checks just the first USB device it finds and ignores the rest.

Problem: booting starts, but after a while, message "end_request: I/O error, dev fd0, sector 0" is displayed periodically.
Solution: The Live USB is trying to find the floppy drive. Switch off the floppy drive in BIOS. In my case, this is done in menu Standard CMOS Features. Set option "Drive A" to "None".

I hope this helps if you have similar problems.


Simple ICQ Export script
2009-12-22T12:42:00.003+01:00
Recently my girlfriend finally decided to stop using the original ICQ program, which she retained mostly because of the unimaginable loss of conversation history. Quick search did not reveal any good and simple means to convert the history data. Most of the software was outdated or for different versions.

We had two versions of history files - one was directory with XML files and the other seems to be Access mdb (created by ICQ version 6 I think). To sharpen my Python skills (which at the time being is a bit like trying to sharpen a club), I decided to write simple conversion script for the XML history files.

The script is published on http://code.google.com/p/icqexport/. For convenience, I have also prepared Windows build http://icqexport.googlecode.com/files/icqexport_01.zip.  You are free to do anything with the script and welcome to send me notes and suggestions.

The usage is very simple, although nothing is guaranteed :) Copy the script to the directory where ICQ history files are stored and run it. It will traverse all subdirectories and convert history XML files to one readable TXT file.

The only version supported is 0x00140034 (tested on ICQ version 5.1). You're fine if there's something like this at the beginning of your files:

<root>
<version>0x00140034</version>
<event>
...

I didn't see the mdb structure yet but if I will get to it, I'll keep you informed.


Tomix - First level ready
2009-05-11T18:23:00.005+02:00
During this weekend I planned to finish the first version of first level of our puzzle game. I had kinda slacking mood (caused partially by this great blog), but I forced myself to overcome it and in the end I was able to finish the first level as planned. There are still no graphics so I'm reusing arrows from Heartbeat, but the basic "engine" is ready. There is still a lot of work to do but being able to actually play the level is really motivational.

I handed it over to Mitsuki to try the game logic and she quickly found a solution - although not the one I had originally envisioned :) But it was possible because of a bug in checking of the final position so in the next version it won't count. But at first glance it seems that the idea is viable and I have a lot of great feedback for the game from Mitsuki. We only have to stay focused on the fact that we create puzzle game, because some ideas would lead us to completely different genre :)

So now Mitsuki will start working on the graphics to incorporate to the game. In the evening I also checked some sites with free game development resources so we could re-use something already created, but my search did not finish yet. But I have a few good sources I want to check so expect another post covering those I think are most useful.

For the next work I hope to finally start work on the editor. I kinda didn't get to that yet, but I already know how to serialize objects in Python :) And it has not that high priority for me - worst case we can easily define the levels in source code.

I also checked some libraries to create menus in pygame but unfortunately none of them seem appropriate to what I want to do so I'll probably have to write my own solution. KezMenu looks fine but it can't center the text and unfortunately it's GPL'd - and I did not decide on the license to code yet, so I may not be able to use that. Well, we'll see.


Device identification problem during boot
2009-05-02T10:34:00.007+02:00
Since installing xubuntu last autumn, it sometimes happened to me that my machine required several restarts to boot up. I have identified the problem but being linux n00b I was unable to find some solution to it for some time.

The problem was following:
I have two harddiscs in my computer - one is old IDE disc and the other is new SATA II disc. The Linux system is installed on the new disc which is usually identified as /dev/sda (where /dev/sda5 is the root partition). The other disc is usually on /dev/sdb. What sometimes happened was that the devices where identified the other way round  - IDE being /dev/sda and SATA being /dev/sdb. After that the computer of course did not boot and ended in some sort of fallback console (sorry, I forgot the mode's name).

Sadly, asking administrators at work did not help me (most probably because I was not able to describe the problem correctly - the only thing I found out was that the IDE disc should be /dev/hd something and not /dev/sd something - very helpful).

Before some time I found what I thought will lead to solution, but I was hesitant to change the fstab because I didn't want to break it any more than it was :) What I needed was to identify the mount points by partition label instead of device (see http://tldp.org/HOWTO/html_single/Partition/#labels).

Today, I finally got to fixing it (at least, it appears to be fixed now :)). The whole process involved:
installing GParted which unfortunately did not help me with setting the partition labels and thus was not that useful (it provided me with nice overview of the partitions :))
setting label to ext3 partitions using tune2fs (e.g. tune2fs -L pubsw /dev/hdb1 sets label pubsw to par
setting labels to NTFS partitions using ntfslabel (part of ntfsprogs package)
modifying fstab to use the labels (see http://tldp.org/HOWTO/html_single/Partition/#labels for example fstab)
To sum it up - if you have this problem with identifying devices during boot up, use labels to identify them in fstab. This seems like a good idea generally because as I understand it the device identification can change when switching hardware (imagine adding shiny new HDD to your computer and not being able to boot up as the result - ouch!)

In the end the changes where really not that hard and if I knew a little about Linux I could've fixed this long time ago. But I'm a little smarter now and maybe after some time I will know Linux as good as Windows. Most probably after Windows 7 comes out - because last version I had is Windows XP, I will know very little about both Windows and Linux.


Development update
2009-04-20T19:45:00.004+02:00
I did find a few hours to quickly whip up the first prototype of Tomix (which is the codename of the game I'm working on) and it's ready for me to play with :) Fortunately pygame is really very easy to use for what I'm planning to do and I'm also slowly getting used to vim and Python. But I'm still not very good with either of them.

Back to the prototype - now I can move the player around on the screen and push things (nothing groundbreaking yet, but what would you want for a few hours worth of programming).

I plan now try to design one level to see if my idea of combining Atomix and Sokoban is at least partially viable. Afterward I'd like to start work on the prototype of level editor which will probably take me more time - I need to learn how to work with mouse in pygame and think about reasonable way to save the edited levels (not sure how to serialize things in Python). Hopefully I can find at least one free weekend to actually implement something (all the codes till now were written on train and that is unfortunately not the ideal development environment).

Also, Mitsuki volunteered to help me with the graphics (so the size of development team has doubled :)). This means that if we eventually manage to have some game ready it will look a lot nicer than if I did it on my own. On the other hand, this also reduced number of readers of this blog that are not working on the game from 1 to 0. Maybe I should start looking for some interesting topics to write about :) (Well, I have a few, I just need to find time and mood to try something new)


Slowing down quickly
2009-04-08T20:52:00.005+02:00
It seems that with the spring beginning, I will have less time than I originally expected. Nevertheless, I thought about the story of the puzzle game, which is planned to be roughly based on Atomix and Sokoban.

The story is not very important for this kind of game but it gave me important ideas on which to base the following puzzle. I'm not planning it to have some big impact like in PuzzleQuest (how could I compete with that?) so I will be creating (almost) pure puzzle game.

With the story boiling in my mind, I can now start looking for some free resources to use in the game (mainly music, sounds and probably some textures) and sketch the basic look & feel of the game.

I will also start working on some early prototypes with pygame because I'm still not very good with it (or Python to be honest). I hope to get good insights to the techniques I should use in the final game.

But, as I said, I expect the work to go kinda slowly for now, mainly because I won't have time during this and next weekend. On weekdays I tend to be a little too tired from regular work, but I'll try to do something when on train to work, so maybe I will move forward at least a little.


2009-04-01T18:16:00.004+02:00
Well, it seems that this blog didn't go very well. Or, it went exactly as expected - abandoned after first post. But after a time I have decided to try to write something again, maybe this time it will be better than on the first try.

I began to be quite excited of game programming after my girlfriend and her team finished visual novel Heartbeat for a ren'ai visual novel competition (warning: unless you speak Czech, you probably won't enjoy this game very much). As a little bonus, I programmed very simple DDR simulator for the game in pygame.

Because the team plans to create sequel to Heartbeat (where I should provide some simple puzzles), I decided to improve my Python and pygame programming skills and write some simple game on my own.

I'm still not 100% sure but I will probably be going for some Atomix inspired puzzle game or simple implementation of Alquerque with AI opponent. I hope to write something more after I decide and actually begin implementing something.

So, if anyone is actually reading this, wish me luck and determination to keep at least one of these new projects :)


Brave New World to Explore
2008-06-02T19:50:00.000+02:00
Programming in C-like languages for some time (not very long compared to others, I admit), I have recently "discovered" new programming world that filled me with excitement and joy of programming again.

For some time, I am trying to improve my programming skills by learning not only about new language features of mainstream languages but also by looking at other, not so common ones.

Last year I was coping with Common Lisp and, after wading through Practical Common Lisp, I was able to even solve some minor problems on Project Euler. However, although the language is certainly interesting, I had to force myself to play with it (partially due to strange syntax, or lack of thereof and partially due to my search for usable free IDE). Even so, the experience gave me some new insights to the craft of programming and my trip to the lisp world was very interesting :)

This year I was planning to venture even further to the world of functional programming and began with Haskell. But my choice of introductory material was an unhappy one. Write yourself a Scheme in 48 hours is interesting book, but very hard without previous knowledge of Haskell and I didn't make it far. I played with Haskell without access to the Internet and it really didn't work. But I will certainly return to this book after I digest some easier Haskell book (maybe A Gentle Introduction to Haskell).
So, what is this new exciting world, you ask? Well, it's new just for me, but this new world is world of Python. The language is somewhat familiar and does not contain so foreign concepts as lisp or Haskell, IDE is just about quick install and you can begin to work. And the vast array of libraries! No wonder Python is being called to come "with batteries included." Great benefit to studying and embracing Python is also Mark Pilgrim's awesome book Dive Into Python, which is very readable and funny too.

After learning languages with not so big practical use and less advanced "infrastructure", it's refreshing not to be forced to look for long hours how to solve trivialities.

So, what's there to like? First and foremost, the readability. Python was designed with readability in mind and (from the little knowledge I have acquired so far) this design was quite successful. One thing that seems to add to this readability is the fact that whitespace is significant. Thus, the programmer is not only advised, but even forced to behave according to the standard.

Being dynamically typed language, the pythonistas are required to maintain greater discipline then Java or C# programmers. Again a good thing - writing unit tests becomes a necessity and thus further increases the quality of the code. And dynamic typing seems to result in a lot less code than is necessary in C# (and company).

One thing I am a little concerned about is the fact, that a lot of things are just a matter of convention (i.e. naming first argument of instance methods 'self', not-so-private private methods and a few others). But even in C# it's possible to change private variables of classes via reflection, so Python maybe just makes this easier. However, in both languages this practice is despicable :)
All in all, Python once again ignited my desire to learn new things about programming.  Well, my queue of languages to learn is still quite big (and it seems that it will never reduce to zero). In the future I hope to dwell a little into Erlang or once again try my luck with Haskell.

So, if you are yet living in Java/C# world only and want try to look a little further, I think Python is a safe bet - it's a language not that drastically different (or outright hostile) from the others you know ;)

- Note: Virtually everything I am writing about Python is just my feeling I got from playing with it for a while and reading about it on the web. So beware, I am not speaking out of personal practical experience (that will hopefully come later ;)).