conditional extend in buildout

I’m working on some kind of generic buildout configuration file providing predefined sections to be imported in the deployment of Odoo instances for our customers, and by the developers in the team. Some parts of the setup can be customized locally by the developpers. In order to keep the default name for the buildout configuration file, I found the following trick useful for conditionally importing a customization file in `buildout.cfg`:

[buildout:os.path.isfile(‘local.cfg’)]
extends = local.cfg

Misketonic : software project for Ketogenic Diet management

I have started a software development project called Misketonic. The goal is to help my brother and his wife manage my nephew’s ketogenic diet. There does not seem to be any such project, at least not as free software. The Ketocalc tool is free (as in beer) for heath care professionals (and the patients of professionals having an account on the site) and the code is managed by Nutricia a company which also sales Ketocal, which is an very useful product when following that type of diet. So while my primary aim is to help my nephew, I’ll try and make the tool as useful as possible for other families.

The source code is available on bitbucket. As of today, Misketonic is starting to be useful, but still has lots of rough edges and is not really user friendly, especially for non French speakers, as the main food nutriment information comes from the Ciqual database, which provides aliment names in French. I’ll post updates on this blog as this evolves.

The paranoid #! security guide

The paranoid #! security guide

#! is read ‘crunchbang’ as in http://crunchbang.org/

Foreign data wrappers in PostgreSQL

Foreign data wrappers in PostgreSQL

In 2003, a new extension called SQL/MED (“SQL Management of External Data”) was added to the SQL standard. It is a standardized way of handling access to remote objects in SQL databases. In 2011, PostgreSQL 9.1 was released with a great support of this standard.

In a nutshell, you can now use various Foreign Data Wrappers (FDW) to connect a PostgreSQL Server to remote data stores.

Plumbum

Interesting project Plumbum:

* Tomer Filiba’s blog post introducing Plumbum
* Plumbum documentation

Similar in goal to some of the modules living under the logilab.common package, with a nice and modern API (a lot of things in logilab.common were written before decorators were introduced… Heck, I started working on logilab.common at the time of python 1.5.2…)

setting default value in a python dictionary

I encountered the following idiom in some OpenERP module recently:

some_dict['key'] = some_dict.get('key', {})

It struck me as an unusual way of setting a default value in a dictionary. So I investigated a bit to compare the performance of the construct compared to other ways of doing the same thing, in order to check if what I would have done (f2 below) was the most efficient way in addition to be the (in my opinion, at least) most readable. I used the timeit module and a few different dictionaries in case the presence of the key influenced the performance of the operation:

import timeit

def f1(d):
    d['value'] = d.get('value', {})

def f2(d):
    if 'value' not in d:
        d['value'] = {}

def f3(d):
    d.setdefault('value', {})

if __name__ == '__main__':
    for d in ('{}', "{'a': None, 'b': None}", "{'a': None, 'b': None, 'value': None}"):
        print d
        for i in (1, 2, 3):
            print "f%d" % i
            t = timeit.Timer('f%d(d)' % i,
                             'from __main__ import f1, f2, f3; d = %s' % d)
            print t.repeat()

With my local version (python 2.7.3), f1 is always the slowest, closely followed by f3, and the readable explicit version (f2) also runs 2x faster.

Opening

To get started, I imported a few old blog entries from the time where I worked at Logilab, and used to blog on the company’s web site. Unfortunately, when I left, my account on the web site was wiped out, which resulted in my posts no longer being attributed to me.

pdb.set_trace no longer working: problem solved

I had a bad case of bug hunting today which took me > 5 hours to track down (with the help of Adrien in the end).

I was trying to start a CubicWeb instance on my computer, and was encountering some strange pyro error at startup. So I edited some source file to add a pdb.set_trace() statement and restarted the instance, waiting for Python’s debugger to kick in. But that did not happen. I was baffled. I first checked for standard problems:

• no pdb.py or pdb.pyc was lying around in my Python sys.path
• the pdb.set_trace function had not been silently redefined
• no other thread was bugging me
• the standard input and output were what they were supposed to be
• I was not able to reproduce the issue on other machines

After triple checking everything, grepping everywhere, I asked a question on StackOverflow before taking a lunch break (if you go there, you’ll see the answer). After lunch, no useful answer had come in, so I asked Adrien for help, because two pairs of eyes are better than one in some cases. We dutifully traced down the pdb module’s code to the underlying bdb and cmd modules and learned some interesting things on the way down there. Finally, we found out that the Python code frames which should have been identical where not. This discovery caused further bafflement. We looked at the frames, and saw that one of those frames’s class was a psyco generated wrapper.

It turned out that CubicWeb can use two implementation of the RQL module: one which uses gecode (a C++ library for constraint based programming) and one which uses logilab.constraint (a pure python library for constraint solving). The former is the default, but it would not load on my computer, because the gecode library had been replaced by a more recent version during an upgrade. The pure python implementation tries to use psyco to speed up things. Installing the correct version of libgecode solved the issue. End of story.

When I checked out StackOverflow, Ned Batchelder had provided an answer. I didn’t get the satisfaction of answering the question myself…

Once this was figured out, solving the initial pyro issue took 2 minutes…

Launching Python scripts via Condor

As part of an ongoing customer project, I’ve been learning about the Condor queue management system (actually it is more than just a batch queue management system, tacking the High-throughput computing problem, but in my current project, we’re not using the full possibilities of Condor, and the choice was dictated by other considerations outside the scope of this note). The documentation is excellent, and the features of the product are really amazing (pity the project runs on Windows, and we cannot use 90% of these…).

To launch a job on a computer participating in the Condor farm, you just have to write a job file which looks like this:

Universe=vanilla
Executable=$path_to_executabe
Arguments=$arguments_to_the_executable
InitialDir=$working_directory
Log=$local_logfile_name
Output=$local_file_for_job_stdout
Error=$local_file_for_job_stderr
Queue

and then run condor_submit my_job_file and use condor_q to monitor the status your job (queued, running…)

My program is generating Condor job files and submitting them, and I’ve spent hours yesterday trying to understand why they were all failing : the stderr file contained a message from Python complaining that it could not import site and exiting.

A point which was not clear in the documentation I read (but I probably overlooked it) is that the executable mentionned in the job file is supposed to be a local file on the submission host which is copied to the computer running the job. In the jobs generated by my code, I was using sys.executable for the Executable field, and a path to the python script I wanted to run in the Arguments field. This resulted in the Python interpreter being copied on the execution host and not being able to run because it was not able to find the standard files it needs at startup.

Once I figured this out, the fix was easy: I made my program write a batch script which launched the Python script and changed the job to run that script.

UPDATE : I’m told there is a Transfer_executable=False line I could have put in the script to achieve the same thing.

Why you shoud get rid of os.system, os.popen, etc. in your code

I regularly come across code such as:

output = os.popen('diff -u %s %s' % (appl_file, ref_file), 'r')

Code like this might well work machine but it is buggy and will fail (preferably during the demo or once shipped).

Where is the bug?

It is in the use of %s, which can inject in your command any string you want and also strings you don’t want. The problem is that you probably did not check appl_file and ref_file for weird things (spaces, quotes, semi colons…). Putting quotes around the %s in the string will not solve the issue.

So what should you do? The answer is “use the subprocess module”: subprocess.Popen takes a list of arguments as first parameter, which are passed as-is to the new process creation system call of your platform, and not interpreted by the shell:

pipe = subprocess.Popen(['diff', '-u', appl_file, ref_file], stdout=subprocess.PIPE)
output = pipe.stdout

By now, you should have guessed that the shell=True parameter of subprocess.Popen should not be used unless you really really need it (and even them, I encourage you to question that need).