In March, I’ve set up a Redmine application with the Ruby webserver Webrick. Since then, I’ve migrated to Apache, and then the question was: Which Ruby bridge module to use? It’s not that the choice is large, you have mod_fastcgi, mod_fcgid and mod_rails a.k.a. Passenger. I’ve tried the three of them, and only one was a success.

As for the last post about Redmine, I’ve compiled everything (Apache included) in a custom location and I start the server from there (without root rights).

mod_fastcgi

This is an old module, but it should be easy to use.

After setting up the configuration file to get the public/dispatch.fcgi file, Ive started the server (did I say that Ruby does a wonderful job by providing every wrapper file needed for any webserver? It provides cgi and fcgi skeletton files, and of course everything needed for Webrick). Unfortunately, it didn’t work, it misses the rubygem module, although it is installed as I’ve used it to install Ruby on Rails!

So I left mod_fastcgi where it was, in the dust.

mod_fcgid

This module is based on Fast CGI as well, but is more recent than mod_fastcgi. It uses Unix socks for the communications between Apache and the back-end, and in my configuration, the server didn’t seem able to create socks (I didn’t find a reason on the Internet, but I have to say I didn’t look far, and besides Google has not much to say about this issue. And I still have Passenger). So much for this module.

Passenger

This module is supposed to interact directly with the ruby files, and in fact it does. No need to have a dispatch.fcgi file, it calls the server as Webrick does.

I’ve installed rake with a gem, but Passenger setup didn’t find it, I had to add a symbolic link between lib/ruby/gems/1.8/bin/rake and bin/rake. I’ve installed fastthread as well, and I was good to go.

In the configuration file, I load the appropriate modules:

LoadModule passenger_module /web/src/passenger-2.2.2/ext/apache2/mod_passenger.so
PassengerRoot /web/src/passenger-2.2.2
PassengerRuby /web/bin/ruby

Then I can configure the Virtual Host:

NameVirtualHost *:3000
<VirtualHost *:3000>
 
DocumentRoot /src_custom/redmine-0.8.3/public
 
<Directory /src_custom/redmine-0.8.3/public>
AllowOverride None
Order allow,deny
allow from all
Options Indexes FollowSymLinks MultiViews
</directory>
 
ErrorLog /web/logs/error.log
LogLevel warn
CustomLog /web/logs/access.log combined
ServerSignature On
</VirtualHost>

Conclusion

With Passenger, it was really easy to have a working configuration. The only thing I’m still missing is the ability to set the RAILS_ENV variable to select a different environment than the default (production).

Philippe Baucour from the ‘Unniversité de Franche Comté’ sent me an email saying that he was looking for the rare PhD candidate that would be able to do numerical modeling and material science on top of high-quality Python coding. I can sympathise with this quest: it is very hard to find someone who codes well, and if you want on top of that him to be able to do numerical modeling!

If you are not afraid of French, his PhD proposal is below. Contact him for more information. Please don’t contact me, I am drowning under (very interesting) e-mail.

Utilisation du calcul parallèle pour la modélisation fractale d’un stack de type PEMFC.

Proposition d’allocation de thèse
Thème 1.g : Modélisation, simulation et calcul haute performance
Thème 2.a : Energie, procédés, impacts environnementaux, stockage de l’énergie

Responsables au sein du département ENISYS-FEMTO-ST équipe Modélisation
D. Hissel, M.C. Péra, R. Glises, Ph. Baucour.

Les phénomènes qui prennent place au sein d’un stack de type PEMFC sont de nature multi-physiques et multi-échelles. Ainsi le comportement d’un stack complet ne peut être appréhendé dans sa globalité que s’il on intègre des domaines tout aussi différents que :

• Les phénomènes électriques et électrochimiques.
• La mécanique des fluides,
• Les phénomènes de transferts de chaleur et de matières,

L’ensemble de ces disciplines interagissent à des niveaux d’échelles complètement différents : du dépôt catalytique (i.e. ~um) au stack (i.e. ~m) soit un facteur d’échelle d’environ 106. De plus, les constantes de temps des différents phénomènes sont elles aussi très différentes et rajoutent à la complexité du problème.

Il y a énormément d’études portant sur la modélisation des piles à combustibles mais les difficultés énoncées ci-dessus amènent à faire des restrictions soit sur le domaine d’étude (une cellule), la géométrie (1D ou 2D rarement 3D) ou la représentation des phénomènes (modélisation système). De plus, la puissance de calcul nécessaire pour ce type de problème fortement couplé et non-linéaire n’est pas facilement accessible.
Le travail envisagé consiste à développer une modélisation 3D complète d’un stack à toutes les échelles à la fois de temps et d’espace. L’approche envisagée consiste à utiliser un modèle fractal qui puisse se partitionner et s’adapter à l’ensemble des échelles (temps et espaces) présentes dans un stack. La conception d’un code modulaire permettrait à terme de tester certaines hypothèses sur le fonctionnement des PEMFC. On peut citer par exemple :

• La gestion de l’eau et de l’humidification des gaz.
• Le démarrage à froid.
• Le fonctionnement en mode dégradé.
• Le design des canaux d’alimentation en gaz.
• Étude de la durabilité et de la fiabilité par un cyclage numérique.

Le laboratoire (ENISYS) dispose depuis peu d’un cluster de calcul qui permet d’envisager un modèle complet. Il est composé de 8 noeuds de calcul comportant un total de 64 processeurs pour 64 Go de mémoire et un espace disque de 1 To.
L’objectif de la thèse serait de développer un code parallèle qui permettrait de distribuer sur les 64 coeurs un modèle complet. Ce modèle peut s’envisager comme l’agrégation de modèles à différentes échelles :

• Modèle d’Assemblage Membrane Electrode
• Modèle d’écoulement non-conservatif dans un canal (déjà développé)
• Modèle de comportement thermique des plaques bipolaires
• Modèle de comportement électrique

Ces modèles relativement simples individuellement seront regroupés afin de former un modèle complet. La difficulté consiste à agréger les différents calculs à la fois en terme de temps et d’espaces, on parle alors de spatial computing ou de parallel computing si l’on distribue un problème complexe sur plusieurs processeurs. Dans le cas de la modélisation d’un stack PEMFC, le spatial computing est envisageable pour les différents domaines d’espaces mais il faudra recourir au parallel computing pour combiner l’ensemble des modèles et s’assurer de la convergence.
Cahier des charges de l’étude :
•   Définition du stack étudié en se calquant sur les données expérimentales disponibles.
•   Développement des codes de calcul en s’assurant de la compatibilité avec un fonctionnement dans un cluster.
•   Développement d’un modèle maître faisant la collecte des différents modèles.
•   Définition du partitionnement spatial et temporel.
•   Validation sur des données expérimentales disponibles au laboratoire.
Matériel et logiciel envisagé :

• Utilisation du cluster sur une base de 32 processeurs en utilisation récurrente et 64 processeurs en utilisation intensive
• Programmation en Python des codes individuels et du code maître en utilisant au mieux les bibliothèques de calcul scientifique (Scipy, Numpy, FiPy, PyPar). L’utilisation d’un code propriétaire entraînerait un surcoût exorbitant en termes de licences (64 licences Matlab par exemple !)
• La parallélisation se fera par l’utilisation du MPI (Message Passing Interface) implémenté en Python.
• L’utilisation d’une solution de parallélisation est envisageable à travers l’utilisation de Ipython.

Contact:

                                                                 Dpt-ENISYS

                                                        Energie, Ingénierie des Systèmes

                                                                   multiphysiques

                                                                        Daniel Hissel

                                                        Techn­Hom, 90010 Belfort Cedex, FRANCE

                                                                 Phone : 33 (0) 3 84 58 36 21

                                                                  Fax : 33 (0) 3 84 22 27 22

                                                              @ : danieL.hissel@univ-fcomte.fr

   Franche-Comté Electronique Mécanique Thermique et Optique - Sciences et Technologies

                                                   UMR CNRS 6174

Contact : Monsieur Daniel Hissel

Chef d'équipe Modélisation

I recently sent on a mailing list a few thoughts object-oriented design, so I might as well also be ridiculous on my blog.

I find that in object oriented design, there are two kinds of objects:

• A first kind is the object encoding logics. This is an object for which clever and complex design will hold together the logics of a state-full application. It can often be part of a forest of objects that are linked together via design patterns. The interfaces of these objects are driven by their active role in the application. These objects are prominently present in interactive application and interactive application. They are mostly particular to an application or a framework, and are mostly implementation-defined.

• The second type of object is a data container. It strives to expose a data model that can be of use in various situations, as it is the link between different parts of the code that do not talk to each other apart through data. It is responsible for loose coupling (something that is very important to achieve a maintainable code base) by having a light and shallow interface. It must be interfaced-designed, rather than implementation-designed. One should very easily get a grasp, an almost physical feeling, for the object by simple interaction with it. I have what I call the ‘explaining test’ for these objects: can I explain fully and completely to somebody what the object does, and any possible caveat, without being sidetracked into special discussions? If not, back to the drawing board: the object will not gain acceptance. In my experience, only the objects of the second kind can easily be shared between different projects.

There is no official C++ standard, unlike several languages (Java, Python, …) where there are referentials for code and design style, good practices, … It didn’t exist until this book where two world-renowned C++ authors set the basis for your every day development.

101 coding standards, numbered from 0 to 100 (an echo to the fact that C++ starts counting from 0), this is the content of the book.

Content and opinions

The standards are split in several groups, from policy to type safetiness. Each time, the coding standard is stated, with a short summary and then a discussion. There can be an example if needed, and some references. The standard is always simple enough to follow, and the explanation is clear yet complete.

The handled issues are very vast, oriented towards common pitfalls. Use inheritance when needed, use collaboration elsewhere, do not inherit from a class that isn’t made for inheritance, … When you are used to these pitfalls (because a lot of C++ gurus talk about them in their forum posts, mails or blogs), you may sometimes forget them and write code that is not optimal (in several ways, performance or maintenability). The book is in that regard a good way of having the good practices classified by topics and easilly accessed: you don’t have to check or search on the Internet. Finally if someone has a question on why you used a specific coding standard, you can give a full explanation and a context (and spread the good pratices).

Conclusion

C++ Coding Standards are sometimes more a question of style than of language, but they are part of the general pieces of advices one should follow. C++ is a language that permits a lot of things, perhaps too much, and this set of rules makes it possible to write readable, efficient, robust code.

C++ Coding Standards: 101 Rules, Guidelines, and Best Practices (C++ In-Depth Series) (Paperback)
by Herb Sutter, Andrei Alexandrescu
ISBN: 0321113586

Price: USD 37.26
52 used & new available from USD 22.00

| 4.5 | 27

Last October I added a toolbar for Chaco plots. It was functional, but it wasn’t very pretty. I decided to rewrite it from scratch, with emphasis on improving the appearance and improving the auto-hide feature.

The new toolbar also employs a new feature to Enable: gradients! Gradient support is still a work in progress, but improving daily.

Greetings,

The conference committee is extending the deadline for abstract
submission for the Scipy conference 2009 one week.

On Friday July 3th, at midnight Pacific, we will turn off the abstract
submission on the conference site. Up to then, you can modify the
already-submitted abstract, or submit new abstracts.

The SciPy 2009 executive committee

• Jarrod Millman, UC Berkeley, USA (Conference Chair)
• Gaël Varoquaux, INRIA Saclay, France (Program Co-Chair)
• Stéfan van der Walt, University of Stellenbosch, South Africa (Program Co-Chair)
• Fernando Pérez, UC Berkeley, USA (Tutorial Chair)

We see our EPD Webinar sessions as a great venue for us to provide subscribers with personalized support. Is there a particular challenge you’ve encountered while using EPD? Do you feel like it would be helpful for us to walk you through a process? We encourage you to submit your questions ahead of time so that we can prepare materials and demos to meet your needs.

The webinar format enables us to respond to your questions (either by chat or VOIP, depending on your preference) and share our screen to provide examples and demonstrations.  We feel that this could become an invaluable channel of communication for EPD users, and are excited to see how it progresses.

July’s webinar will be held next Thursday at 1pm to accomodate Independence Day weekend. We plan to give an overview and demonstration of parallel processing with iPython, as we’ve seen the tremendous utility of this EPD feature overlooked in the past. Once again, however, if you have a special topic that you’d like to have addressed, feel free to write us an e-mail to tell us what content you’d like to have covered in Thursday’s session.

EPD Webinar: Thursday July 2, 2009
1pm CDT/6pm UTC.
Register at GoToMeeting.  A password to enter the webinar will be provided in your confirmation.

I bought this book as soon as it was published, and I sold it soon after. Suffice to say I had a very mitigated impression after reading it. There are good things in it, but also some very bad stuff. It doesn’t describe how to write your ultimate game engine, but the author’s game engine. What about some modesty?

Let’s start with the bad stuff.

Content and opinions

This book does not show the engine architecture, the only thing you can see are some pictures with some classes. There are hints about UML, but those pictures are far from being UML. Besides code quality is really disappointing. No const-correctness, no std::string for the parameters, char* are used and then converted to strings (!!!) Really bad coding practices. Another thing is class instantiation before they are actually used whereas the text says it shouldn’t be done! Finally, the book has plenty of code pages, whereas the code is in the CD or on the Internet, so why using so much pages for something that is easily available?

For the good things, the book covers almost every aspect of a multiplatform game engine. For inputs to physics as well as graphism (OpenGL and DirectX), AI, the handling of these systems is there. If some parts show only abstraction for replacable libraries (as for graphism), other show the actual implementation, like for physics. Speaking of physics, it uses a good part of the book to explain how it works, even if it is not a fully-fledged physics engine at the moment of writting.

I regret the fact that the book is about a uncomplete and evlving engine. I also regret the script engine part, because it is kind of false (compiled or interpreted, a script language can always use a virtual machine) and pages could have been better spent for exposing the engine architecture…

Conclusion

As a conclusion, this book was the first (to my knowledge at that time) complete book on a game engine. There are more complete books on parts of it (mainly 3D engines), but nothing on a complete engine. For this, it desevres some credits. Unfortunately, the drawbacks are too big to consider this book a viable option.

Ultimate 3D Game Engine Design & Architecture (Charles River Media Game Development) (Paperback)
by Allen Sherrod
ISBN: 1584504730

Price: USD 37.77
25 used & new available from USD 36.00

| 3 | 5

We are finally opening the registration for the SciPy 2009 conference. It took us time, but the reason  is that we made careful budget estimations to bring the registration cost down.

We are very happy to announce that this year registration to the conference will be only $150, tutorial $100, and students get half price! We made this effort because we hope it will open up the conference to more people, especially students that often have to finance such trip with little budget. As a consequence, however, catering at noon is not included.

This does not mean that we are getting a reduced conference. Quite on the contrary, this year we have two keynote speakers. And what speakers: Peter Norvig and Jon Guyer! Peter Norvig is the director of research at Google and Jon Guyer is a research scientist at NIST, in the Thermodynamics and Kinetics Group, where he leads a fiPy, a finite element project in Python.

The SciPy 2009 Conference

SciPy 2009, the 8th Python in Science conference, will be held from August 18-23, 2009 at Caltech in Pasadena, CA, USA.

Each year SciPy attracts leading figures in research and scientific software development with Python from a wide range of scientific and engineering disciplines. The focus of the conference is both on scientific libraries and tools developed with Python and on scientific or engineering achievements using Python.

Call for Papers

We welcome contributions from the industry as well as the academic world. Indeed, industrial research and development as well academic research face the challenge of mastering IT tools for exploration, modeling and analysis.

We look forward to hearing your recent breakthroughs using Python! Please read the full call for papers.

Important Dates

• Friday, June 26: Abstracts Due
• Saturday, July 4: Announce accepted talks, post schedule
• Friday, July 10: Early Registration ends
• Tuesday-Wednesday, August 18-19: Tutorials
• Thursday-Friday, August 20-21: Conference
• Saturday-Sunday, August 22-23: Sprints
• Friday, September 4: Papers for proceedings due

The SciPy 2009 executive committee

• Jarrod Millman, UC Berkeley, USA (Conference Chair)
• Gaël Varoquaux, INRIA Saclay, France (Program Co-Chair)
• Stéfan van der Walt, University of Stellenbosch, South Africa (Program Co-Chair)
• Fernando Pérez, UC Berkeley, USA (Tutorial Chair)

Update: I correct the typo in the original blog post: the sprints are free, the tutorial are $100.

What an appetizing title! This book is part of an O’Reilly serie that treats a lot of interesting topic. Contrary to Beautiful Code, this one is much shorter but the title suggest it is much more pragmatic.

Each time, the author has some thing to say, it starts with a small phrase that can be disturbing, and then a complete explanation follows. This means you should be ready to be shaken, what the author tries to teach you are sometimes things you have to think about twice (at least I had this feeling).
The book is plit in two parts, theory and pratice. The first one is the longest, with 4 chapters, still the second has more chapters, with 10 ones.

Content and opinions

So the first part is about theory. A lot of things make sense, and even if you didn’t read the book, you should know them. Acceleration is about how you can do things faster. Well, it’s not much about developing than setting up you work environment (search utilities, shells, …) so that you can access things faster. Focus is about maintaining focus during development. Less distractions so that you can keep be focused (you need 15 minutes until you achive maximum productivity, so if you’re interrupted every 5 minutes, it’s cumbersome), more space on your desktop with several monitors or virtual desktops (the book gives links to useful tools, like virtual desktops for Windows), … Automation is linked to acceleration, but it is more, it is about doing the same thing several times. Different tools (bash, ruby, …) are used in different examples. What I mainly remember is that you need to master several different tools. The last chapter is Canonality, or the DRY principle (Don’t Repeat Yourself). There are several cases where you seem to need to have several times the same things (when maintaining UML diagrams, for instance), and then using automation, you can have canonality. This is spreading through several pages, and although it is natural to use automation to achieve canonality, you have to realize it first.

The second part is about practice, and more exactly what tools and processes you should use, based on the author’s experience. It seems to be mainly based on the agile patterns, with a start on test-driven development, and then several topics on software architecture, and they come regularly back in following chapters (a good thing INO). Two chapters are about the old times, one dedicated to what good can be extracted from past experiences, and one dedicated to bad experiences that keep on harming development (mainly people that think they know the truth, without questionning themselves, the so-called angry monkeys). The last two chapters are about using the right language for the application and using the right tool (the IDE in this case) for maximum productivity. This part about the actual ways of being more productive achieves, IMHO, its goal with good impulses for a productive programmer.

Conclusion

A lot of the tools that are presented in the first part are not free or open source, and sometimes there exists alternatives in the free community, so you have to look for them. The secon part tends to look more towards those free tools.
The whole book is definitely about good practices, with some more controversial than others. Some of them are also difficult to apply to any language, as tools for this practice are missing. I think the book describes a good set of practices to try to apply personnally ; at least, I’ve decided to try to and to better use the command-line and my IDEs (also fight angry monkeys).

The Productive Programmer (Theory in Practice (O’Reilly)) (Paperback)
by Neal Ford
ISBN: 0596519788

Price: USD 26.39
53 used & new available from USD 19.75

| 4.5 | 20

Fantastic:

I had such a blast at the last public webinar that we did to promote Python for Scientific Computing. I am really looking forward to the next webinar which is only a week away (June 19). We had 100 people attend the last one. I know that some who wanted to attend could not because of a mix-up on times, or a problem with the fact that GoToMeeting doesn’t support Linux (I’m not very happy about that, but I don’t see another option right now). I apologize for all those problems, but hope you will try to attend again.

There is a lot that we could cover in these webinars, and I’m anxious for your feedback about what you would like to see. My plan is to put a schedule together so the topics are listed through the end of December after this next webinar. Now is the chance to make your opinion known if you’d like to steer these webinars in a particular direction. Schedules are busy and varied, so I’d like to give plenty of notice so that more people can attend the webinar they are most interested in.

In this upcoming webinar we are going to provide an introduction and demo of Chaco (which we didn’t get to the last time). If there is time, I will also continue the Mayavi demonstration (particular the mlab interface) that we started last time, but I also wanted to showoff EPDLab to a wider audience. You can register for the webinar at https://www1.gotomeeting.com/register/303689873

In the EPD subscriber webinar on June 5th, we discussed EPDLab (an open-source interactive Python environment included as part of EPD). Because EPDLab is a free and open-source project that anyone can participate in, contribute code to, and use as they would like, I think it deserves some attention at this next public webinar. Not only does it provide an enhanced scientific computing environment, it also provides an introduction to the Enthought Tool Suite (a free and open-source collection of tools for building compelling scientific applications — it goes by the abbreviation ETS).

I hope you will excuse a brief aside to clarify ETS and its relationship to EPD. Because we do sell a binary distribution of Python tools called the Enthought Python Distribution (EPD — which also happens to contain ETS), there is sometimes some confusion regarding the license and availability of ETS. ETS is a large BSD-licensed open-source collection of tools with a public SVN repository that anyone can contribute to and participate in the development of. Enthought has released a lot of code in that library which has made it possible for us to write sophisticated, compelling, and attractive scientific computing applications for our customers. ETS contains multiple separate projects. The most important and developed of these projects are Mayavi, Chaco, Traits, TraitsUI, and Envisage. You can learn more about ETS at Enthought’s open source portal.

But, Enthought is a small company and the majority of our marketing effort right now is centered around getting the word out about EPD and our other products and services like training and custom software creation. We don’t have the man-power to advertise ETS very well at the same time, and it can be a little confusing that EPD the distribution does cost money for commercial use, but ETS is free and open-source. Fortunately, people like Gael Varoquaux and Prabhu Ramachandran lead the internet charge to spread the word about the great tools in ETS.

I’m looking forward to seeing many of you on-line again at 1:00pm (Central Daylight Time) on Friday, June 19th. Slides and a recording of the webinar will also be made available here after conclusion of the webinar.

Since this post, Intel has officially released Parallel Studio. This is why I’ve published a new, up-to-date review here.

Our lab is seeking to hire an engineer to work on porting our machine learning code to the scikit learn, adding tests and documentation and packaging it.

We are looking for someone motivated by quality in software and open source. No prior scientific computing experience is required. You will be working in a highly stimulating research environment (Neurospin), near Paris and employed by the French research institute in computer science and applied math (INRIA), a prestigious institution.

Neurospin is a research institute dedicated to the understanding of the brain. You will be working with computer-assisted neurology laboratory, the image-analysis and branch of Neurospin, in the small ‘Parietal’ INRIA team embedded in NeuroSpin and dedicated to statistical modeling.

Over the years, the lab has developed a set of tools for machine learning and statistical analysis in Python (with some C). There are some tools for this purpose available in the open-source world (BSD-licensed) in the scikit learn. We want to extract the good and unique parts of our internal library, and release it in the open source world through the scikit learn. Our code is fully Python code, using scipy and matlab, with some bindings to R. As we want the code to be BSD-licensed, we will remove the bindings with R, and replace when possible. The job does not involve developing new algorithms, but testing, improving, and documenting the existing one. There is a big quality assurance work to be done. The code needs to be put to the right coding standards; APIs should be cleaned; tests added. Dead code should be delete. There is some optimization work to be done. Also, if there is any duplicated funcitonnality with the scikit learn, you should analyse both code and determine which one to code. The job also involves working with the community, documentating the code, and releasing the project, including binary packages. And finally, all the original authors of the algorithms, and experts in the field, are in the lab. So you will be able to learn from them and pester them if there is a problem with the code.

In one word, this is about transforming an internal project, into a leading open source project that will rock and live on!

The job description is available here.

There are to caveats: first it is a 2 year position. Second, you need to have graduated recently (how recently I don’t know exactly, but I will inquire).

If you are interested, or just want to ask questions, don’t hesitate to send me an e-mail, I am _really_ looking forward to collaborate with someone motivated on this project.

UPDATE: I have more details on the restrictions of the job offering: you need to have graduated in 2008 or 2009. This is a very hard restriction, and I am recieving many excellent CVs that I even consider because of this restriction. I am sorry, I cannot do anything about it.

I wanted to thank everybody who came to the first EPD Webinar which was held today at 1:00pm CDT in our offices at Austin. We had a few technical glitches which our team resolved quickly. I then spent about 30 minutes showing off new features of EPD such as EPDLab, indexed searching of docstrings with Whoosh, and the new curve_fit function from scipy.optimize. Dave Peterson then spent about 30 minutes showing the use of enpkg which is a command in EPD to allow update, upgrade, and rollback service for egg-packages. This tool should allow subscribers to EPD to keep up to date without having to download and install new installers every time a release is made.

A packaging tool like enpkg has been on the roadmap since the beginning, and it was encouraging to see it in action. There are still a few speed and “verbosity” issues that we are cleaning up, but it looks like a good start to what should be a very useful feature for EPD.

In the future, the EPD webinars will contain about 30-45 minutes of training material drawn from the 7-days of course material on scientific computing with Python that we teach regularly. If there are particular points you would like to see covered, please let us know at info@enthought.com. The current plan for the next EPD Webinar is to provide training on the statistical capabilities of SciPy.

All who attended had the chance to ask questions directly of the Enthought attendees. We look forward to answering more of your questions in the future. The next EPD webinar is scheduled for Friday, July 3 at 1:00pm CDT. If you are an EPD Subscriber at the Basic level, please register. We look forward to your attendance and questions. Feel free to send pre-webinar questions to info@enthought.com

The next public webinar for general scientific computing with Python is
Friday, June 19 at 1:00pm CDT. This webinar is open to all that would like to attend. Right now the plan is to show-off the open-source EPDLab and give an overview of all the tools that are brought together in EPD. You can sign up now for the event. I look forward to seeing many of you attend.

We’ve recently made the decision to start applying resources to generating x86_64 OS X builds of EPD.  Because of limited resources, this means we’re officially dropping PPC support in EPD.  It also means that it may take us months to get things released for the x86_64 (also known as amd64) architecture.

As an example of some of the issues we’ll face, we’ll need to decide how to handle the GUI backend situation.  You see, the wxWidgets project hasn’t yet released 64-bit build support for OS X’s Cocoa framework, and the Carbon framework isn’t 64-bit, so we’re stuck either starting with a “server” / console build of EPD, shipping on an unreleased version of wxWidgets, delaying the release while we help finalize x86_64 Cocoa builds of wxWidgets and wxPython, or switching to a different backend like Qt.

While Qt and the PyQt (Python bindings for Qt) seems like a no-brainer technology-wise, the license situation is a hurdle for us to overcome.  We’ve tried hard, but haven’t always succeeded, to avoid GPL licensed projects in EPD in order to make it more palatable to commercial users, like even our own consulting projects.  And, yes, Qt itself recently came out with an LGPL license option that would suit EPD’s needs, but PyQt isn’t similarly licensed (yet).  So now we have to decide whether such a core capability (the only GUI backend of OS X x86_64) would be acceptable to be GPL licensed.

If any one has any thoughts or suggestions on how to resolve this issue, please don’t hesitate to let us know!

By the way, regarding the PPC situation, we have effectively already started to drop PPC support with the EPD Py25 v4.3.0 release.  We made a good faith effort to build in the PPC support but simply didn’t do significant testing of the results.  In the end, it turns out that at least one core module, SciPy, ended up with binaries that don’t fully support PPC.  Sorry, but we do not plan to issue fixes for this.

If you’re a PPC user, the last working version of EPD for PPC was EPD Py25 v4.2.30201.

Strategy games are the type of games I prefer. Turn-based or real-time, they share some common ground. This book tries to explain them.

Content and opinions

The book is split in three parts: war, peace and design.

War is generally at the center of a strategy game (war, or tensions that can lead to it). War consists of units fighting together. Equilibrium is something that is not easy to achieve, and the game must suggest different strategies with those units to remain interesting. Relevant facts about those topics are introduced.

The Peace chapter is not only about peace, perhaps only truce before the war. It’s about resources, technologies or stuff like that. It is not mentioned that too many different resources can lead to a bad game as well (too complicated to handle, an AI that cannot cope with it can ruin the game, …)

Finally, almost half of the book is dedicated to design. Not only interface design, but also game and gameplay design. Do I need a hero? How far does the world strech? These kind of questions can make really different games, and also a game that wanted things to big can in the end be a failure because of these.

Conclusion

This small book can be read in a few hours. It is mainly based on actual game images, but their legends are too small, especially when they convey useful information. This problem put aside, the book is really enjoyable, it does not go into too many details, only to give a first overview of what a strategy game should be.

Game Guru: Strategy Games (Paperback)
by Dave Morris
ISBN: 1592002536

Price: —
21 used & new available from USD 8.95

| 4 | 1

I’ve promised to make an update whenever I would find a solution to the problem I had some months ago when I tried to use the latest MKL with numpy. Well, there was a simple hack that did the trick. It is far from being perfect, but at least, the tests pass now.
So the only thing you have to do is to export the LD_PRELOAD variable:

export LD_PRELOAD=/path/to/the/MKL/lib/libmkl_core.so

We’ve just updated the EPD product website with links to a new EPD repository RSS feed: http://www.enthought.com/products/epd-rss.xml.

This feed is updated every time we release an update / upgrade to a project included in EPD, and soon everytime we publish a new installer.  All entries into the feed specify the platform the update / upgrade was released for, the date and time of release, the name of the thing being updated / upgraded, and a short description of the project being updated.

We’ve added this feed to both the main EPD product page, and also the ‘Download’ page — they’re the same feed so no need to subscribe twice!  For those with browsers that recognize RSS declarations in the page/HTML headers, you’ll see the RSS feed icon in your browser’s URL field when visiting these pages.   For others, simply click on the explicit RSS link on the main page right under the big, red, “Download Now” button.

As an alternative to this all-platforms feed which is available to everyone, even those without an EPD subscription at all, current EPD subscribers with access to the repository can subscribe to a filtered RSS feed for their target platform(s).  These can be found within the  various platform-specific directories under the top-level ‘eggs’ directory.  Note that only those with ‘Basic’ subscriptions and above can access the repository.

Please don’t hesitate to send us suggestions on how we can continue to improve EPD!

Enthought’s first public webinar was a success, despite a few technical glitches. The large attendance in spite of the short notice was gratifying. Travis was able to cover only a fraction of the material he had hoped to cover, so there is plenty of material for future sessions (such as Chaco and Mayavi).

The recording of the webinar is now available for download. While the native recording format of GoToWebinar is Windows Media Player, we have converted it to Matroska format, so we hope that folks on all platforms will be able to view it. Please let us know if you have problems getting or playing it.

The next public webinar will be Friday, June 19 at 1:00 CDT. Specific topics TBD.

Meanwhile, we are launching a second webinar series, exclusively for subscribers to the Enthought Python Distribution at the Basic support level or higher.  Those subscribers will receive an e-mail announcement shortly.

IronPython is the first dynamic language developed for the .Net plateform. At first, .Net didn’t support this kind of language. This is something that keeps on coming back througout the book: you have to use some additional tricks to unleash the power of .Net dynamic and static languages.

Content and opinions

The book starts with a general introduction to IronPython. A quick review of the language itself is followed by the use of the .Net assemblies. At the end of this part, one is comfortable enough to do some small IronPython programs.
The next part is dedicated to what IronPython offers thanks to Python and to its .Net affiliation. The authors go through standard Python (battery included) and the somewhat associated .Net assemblies (some arguments on using one or the other could have been a big plus to the explanations), depending on what must be done. Because or (or thanks to) .Net, several pages are dedicated to XML, as it is needed to simplify the description of UIs. Also several useful designed patterns are presented with the .Net approach.
The next part starts with WPF, the official graphical interface, with several ways of using it (bridge from C#, XAML, …). Then WMI (used for system administration) is handled, but from my point of view, it is the weirdest part. WMI has its own language which does not seem like C# or Python. Besides, PowerShell, presented as well as a way of doing system administration, has its own language. There is a book dedicated to PowerShell, so only the communication between IronPython and PowerShell is handled. So two additional languages in this chapter, perhaps too many (they are limited to this chapter).
IronPython is a .Net language, so it is possible to do ASP with it. A chapter deals with this approach, chapter well written but it needs to follow the associated example in your favorite IDE if you want to follow what’s happening. Web means also web services and databases, handled in one chapter. The basis of SQL tools addressed, as well as basic webservces (mainly REST). I have to say that there are some mistakes there, as SOAP is not only used with POST HTTP requests but also with GET requests (it can be seen in the official w3c specification) and also with other transport protocols than HTTP. Perhaps these are .Net implementation’s limitation, in which case it should have been mentioned Finally Silverlight integration allows developping light clients that can interact with other langages as well as the web page.
Throughout the book, complete interaction with other .Net languages was not addressed. It is the goal of the last part to show how assemblies can be used in IronPython and how IronPython scripts can be used from .Net static languages. As I’ve said, the interaction does not go completely smoothly, there are several solutions to accomplish it. At least, the book does not only speak about the upcoming .Net 4.0 that will help this interaction.

Conclusion

As a conclusion, those who need a dynamic language (to script an application) can go for IronPython, th first dynamic language for the .Net framework, compatible with the langage Python 2.5, and in that case, go for this book that will help you for anything.

IronPython in Action (Paperback)
by Michael Foord, Christian Muirhead
ISBN: 1933988339

Price: USD 29.69
44 used & new available from USD 21.50

| 4.5 | 7

We are trying something new at Enthought. I’m going to host the first Enthought Webinar on Scientific Computing. This webinar is free for people interested in showing up. You should plan to come with a bit of patience as we may not have all the wrinkles worked out of the technology and what it means for having a discussion.

I want to spread the word about all the very cool tools that the Open Source community has produced for using Python in Science. The first webinar will be on Friday at 3:00pm CDT. You can attend via your computer by registering at the following link: Python for Scientific Computing Webinar. I will be talking about NumPy, structured data-types, and memory mapped arrays (and how to use them for reading data quickly from files). I will also be showing off Chaco for 2-d interactive visualization and Mayavi for 3-d visualization. Come with questions as you will have the opportunity to ask them if you would like.

We will start 15 minutes early for people who want to get help setting up with the Webinar technology from GotoMeeting. If you have never attended a Webinar before, you may want to come and try it out. I look forward to seeing many of you online this Friday.

Some months ago, I’ve decided to dig into raytracing, and more exactly interactive raytracing. So I’ve started writting my own library, based on several publications.
nVidia announced recently its own framework, Intel wants also to do raytracing on Larrabee, it is the current trend.

First, raytracing is a tool to draw a picture from a scene. It’s like a camera observing the scene, and to know the content of each pixel on the film, a ray is cast through the scene. Depending on the objects, rays will propagate to other objects or lights, and then the actual color of the camera pixel can then be computed.

A raytracer is in fact really easy to write. The math formulae are simple for a version that will return satisfactory results. On the contrary, an interactive or “real-time” one is more complicated. This is why I started my small project and why I will blog from time to time about it.

Interactive or real-time means that the drawing must be fast, really fast. This also means that realistic drawing is out of the question. I’ve started with a simple raytracer, with some optimizations, and then I’ve added some algorithmic optimizations. I will talk about all this in several posts.

How does a raytracer actually work?

In physical vision, light rays travel around the world from the sources, hitting objects, then reflected or transmitted and finally arriving at the eye (or camera). The principle of raytracing is to do it backwards, from the camera to the light sources. The camera film, or the retina, is symbolized by a screen.

How a raytracer works

In this case, the emitted ray hits object 1 with some angle to the normal (the normal vector is the vector orthogonal to the tangent plane of the object). Object 1 gets some light and so the color ray depends on this light, the object and the angle between the normal and the direction of the light.
The ray is then reflected to object 2 and hits it with another angle. As for object 2, it gets some light and the reflected ray carries a specific color. This color is then blended with the color of the primary ray, and this final color is the color of the pixel on the screen.

Coming next

I’ve written my library in C++, tested it with several compilers on different platforms. So as to test the code, to profile it and to create a scene in an elegant manner, I’ve added a small Python wrapper. This will be the subject of my next post.

Then, I’ll go through secondary and shadow rays, a GUI for the raytracer, acceleration structures, …

The code is released under the LGPL on Launchpad.

Additional links

Interactive Raytracer on Intel website

An Intel article on real-time raytracer architecture

nVidia presentation on raytracing

May 30th and 31st the French Python conference, Pycon FR, will be held at ‘la citée des sciences’, la Villette, in Paris.

The first day, I will be giving a one-hour-long tutorial (in French) on numpy, scipy, and all the Python for Science jazz. On the following day, I will be giving a half-hour-long talk to ilustrate the use of Python in my current work: statistical analysis and modelling of brain activity.

I’ll be giving my tutorial in one room, while David Larlet (the famous Biologeek) will be giving one on Django in another room. Tough competition .

The program of the conference is very eclectic, ranging from general programming talks, to GUIs or web development. While this might deter the pure scientific computing folks, I strongly encourage you to attend. Indeed, a lot of the development, packaging, quality assurance, … problems encountered in scientific computing are universal in computing.

You might think that you are only interested in writing algorithms,or processing data, but this code will have to live on. My experience is that it is terribly hard to have code in a lab that can be somewhat shared and live on when people move away to another lab, or stop having time to maintain the code. Talks like

• Correction d’un bug et naissance d’une nouvelle fonctionnalité dans CPython
• Contrôle de versions de source: pourquoi? comment?
• De la qualité dans un projet en Python
• Amusez-vous à tester avec les objets farceurs (mock)

can probably be of some use.

Also, don’t underestimate the fact that some other communities might have solved some of the issues you struggle with. When dealing with real-world problems, and not only developing algorithms on a few set of test data, a large fraction of the code lines and related to IO, interfaces, data massaging… Two years ago, I remember that I was not terribly interested in the web-development talks. I tried to be open-minded and listen to them, but… Now I have done a bit of web development myself, and I have played with some of the famous ‘web frameworks’. I can tell you, there are some really interesting concepts there. The web guys have managed to extract a set of patterns from the problems they face and provide excellent abstracts to data handling and display. Can we learn from them? I am especially interested in getting more insight from things like ORMs (object relational mappers), and understanding better the web frameworks:

• Django-ROA pour une architecture orientée ressources
• PyQt4: Un exemple de sur-mesure en Model/View/Delegate (this is not about web, but MVC/MVD pattern has been used in web a lot and is universal and very important, IMHO).

• Oubliez le sql avec SQLAlchemy
• Developpement d’applications maintenables avec Django
• Turbogears 2, présentation et introduction (tutoriel)
• Programmer CouchDB avec couchdbkit
• Réflexion sur l’utilisation de python pour le développement d’une plateforme web d’annotation génomique
• Oubliez le sql avec SQLAlchemy
• Django par la pratique
• Python et les bases de données non sql.

And finally, one more reason to come: it is so nice to actually get to meet in real life people, and have a chat.

So, see you there, for those who live in France.

A few of us at Enthought (Peter Wang, Robert Kern, and I) traveled to Toronto two weeks ago to attend a very interesting summit of scientists and others connected to finance and economics to discuss whether and how science can provide assistance in understanding economics sufficiently to prevent or at least mitigate economic breakdowns such as the one we’ve just experienced (and are still dealing with). The conference was titled The Economic Crisis and its Implications for the Science of Economics. Some background material for the conference can be read at Edge.org, and at least two blog-posts covering the conference can be read: one by Stephen Hsu and another by Barkley Rosser.

We were invited because Eric Weinstein is a fan of Python and the tools in the Enthought Python Distribution (including NumPy, SciPy, SymPy, MayaVI, and Chaco). Robert Kern produced some very nice visualizations for Eric’s talks in the conference using MayaVi and Chaco which can be seen in Eric Weinstein’s two talks: 30 minutes into the first one and 45 minutes and again 1:30 minutes into the second one (Actually, the second talk was Pia Malaney’s talk and Eric enthusiastically joined her half-way through — I guess being married has its advantages for getting more air time.)

The conference was intellectually stimulating and very enjoyable. I enjoyed all of the conversations I personally had with the participants which ranged from probability theory to cognitive neuroscience to quantum mechanics to computer platforms for agent-based modeling. I encourage you to read and listen in more depth to what the participants had to say in their talks because I won’t be able to provide sufficient summary to the conference. All of the conference talks are online. What isn’t shown in the videos, though, are the break-out discussions that took place between sessions and at meal-time.

In these break-out discussions I enjoyed getting to know all four members of the PartEcon team. Apparently, Mike Brown organized this group after an agent-based model (discussed at the conference by Alexander (Sasha) Outkin) predicted some useful results of changing the tick-size to decimals on the NASDAQ. They have incorporated principles of double-entry book-keeping into their agent-based model. They also stayed after the conference to continue comparing notes with another team from the Perimeter Institute that had written about an agent-based model using a more formal setup (by Samuel Vazquez and Simone Severini).

While there was one early talk on the first day by Richard Alexander that touched on the genetic component of human agents, the impact of having evolutionary biologists present (like him and one of his students, Bret Weinstein) was much larger than their presentation footprint. They provided insightful discussions during several break-out sessions (Peter Wang even commented that in another life he might have become a biologist).

Lee Smolin sent around a very nice summary of the conference and suggested a unifying theme of “path-dependence in economic dynamics.” Eric and Lee were both there to explain how gauge theory provides the tools to solve the problem of changing preferences that has plagued traditional academic economics. Eric did a great job of showing how this manifestly untrue concept of unchanging preferences has at least been put forward by several leading economists. It’s still unclear to me whether or not gauge theory actually provides new results, but it definitely seems like a more useful mathematical toolbox to use and build from.

I was disappointed that amidst all the discussion of the failure of economic modeling there was not at least some discussion about the Mises-Rothbardian ideas of fiat currency and fractional-reserve banking being the primary source of the booms and resulting busts. I wanted to learn from the people there rather than try and debate this one particular theory of economics so I pretty much stayed quiet. One gentleman sitting next to me during the first day asked the panel whether the crises shows the failure of fiat currency and and got a very unsatisfying answer from Nouriel Roubini that simply dismissed the question, but did not really address it.

Given that the economic experts have basically shown repeatedly they don’t know what they are doing, intellectual honesty would seem to me to require listening to all sides of a debate, instead of dismissing a whole theory of economics (such as the Austrian school) primarily because it doesn’t use math as its starting point. Fortunately, there are very good texts that argue against fractional-reserve banking and the role it may actually play in causing economic instability. One of them is “Money, Bank Credit, and Economic Cycles” by Jesus Huerta de Soto.

I really enjoyed the conference because it seemed to combine all of the interests I’ve developed over the years: math, probability theory, neuroscience, economics, and computers. I’ve had a hobbyist interest in Economics ever since graduate school at the Mayo Clinic when I was learning about Linux and Python. I fell in love with open source software but wanted to understand how “giving software away” could work sustainably in a society. It was this question that led to me finally reading Mises and Rothbard and a whole host of other non main-stream economists. I can’t say I’ve figured anything out, but I have very much enjoyed the ride.

I’m also very hopeful in some of the ideas I saw at the conference that may help us inch closer to an understanding of the truth of an economic system (mathematically modeling changing preferences, using agent-based models, and even the idea of local currencies that was discussed among some at the conference).

In the more immediate future. It looks like there is some discussion afoot for building a platform for agent-based modeling that I hope Python plays prominently in. There is a real power in using an expressive and dynamic language like Python that allows for rapid development. It is a general-purpose language that scientists and engineers can actually get excited about. In addition, the work of Paul Borrill’s company (Replicus) in creating an agent-based storage solution looks immediately promising. Perhaps Enthought can provide some tools to assist in managing such a system. I’m enthused and anxious to continue to support the improvement of using computers to help solve some of the world’s most challenging problems. There is much more that could be said, but I’m sure this blog (with no photos) is long enough.

I got my hand on an old edition of this book second edition, now the third is available), and it seemed to me a good place for game developers to start.
Mike McShaffry has a lot of experience from the game field, and his goal is to share it with the readers. In every chapter, there are some anecdots of his past, and it is a lot of fun to see studios falling in the same pitfalls than we do when we start coding.

Content and opinions

The book is split in four different parts. The first one starts with the fun you can get coding a game, but also the troubles you will have. And what technology will you use? 2D? 3D? And what do they imply? As for every code, there is a set of general good pratices, as memory handling, scripts, … that need to be address. The author sometimes did not use them, and there are examples where they caused troubles.

As to get the game running on a computer, another set of rules is needed. Without them, it is just hard to have a running game in the end. How is the game built (not everybody uses a tool to automaticaly build the game)? How to interact with the game? A lot is written about this last issue, and as the author is used to Direct X, the clues are explained with it. But the advice can be used with other technologies. One just has to find the equivalent functions in the other framework. Obviously, it is not possible for the book to express this in every available framework, and it is also not the purpose of the book (it is not a book on a game engine, not a book on Direct X, …).

The third part is also mainly about Direct X, more exactly the 3D part. It lays down the basis for any 3D game engine, but it is not the book’s goal to be exhaustive about the design of a 3D engine. Also Microsoft imposes a set of rules to get the appropriate “Windows compatible” logo, which is needed if you want to sell the game. The last chapter in this part tacklesdebugging the game. I have to say this is much needed and too often it doesn’t appear in game programming books, although it is one of the pillar of programming. Different debugging techniques are addressed.

Finally, the last part tackles how the coding must be driven. Scheduling and milestones, testing and fixing the bugs or how the game will finally be published (what needs to be done at the end or after the end), all you need if you are in the game industry and you have to handle a commercial release.

Conclusion

My edition of the book is several years old, and I felt it, as different examples are outdated (the requirements needed for a today game, the Direct X version, Windows versions that need to be supported, ..). I couldn’t check in the thirs edition if this was updated, but it should have: the whole point of a new edition is to update these facts. So if you want to code a game, buy the last edition of this book.

Game Coding Complete (Paperback)
by Mike McShaffry
ISBN: 1584506806

Price: USD 37.79
31 used & new available from USD 34.14

| 4.5 | 26

Gary Ruben came up with the excellent idea of visualizing the minimum spanning tree of a Delaunay tesselation in addition to Delaunay tessalation itself. After he sent me his code, I spent some times playing with it, because I found out that, with the right choice of visualization parameter, it gave me a nice understanding of what a minimum spanning tree was: a tree structure of minimal total length connecting all the vertices of the graphs, and embedded in the graph. On the visualization, the Delaunay graph is displayed in grey, and the minimum spanning tree in thick and colors.

The minimum spanning tree is calculated using Prim’s algorithms, on the fullly-connected distance-weighted graph of all points. One can clearly see that is it embedded in the Delaunay graph. In fact I have tested that calculating a minimum spanning tree on the Delaunay graph, or on the complete graph, gave the same result.

The code to create this picture can be found here.

Hi Ondrej,

How are you?

I am about to publish a new free software project, a new simple PHP framework, and I am interested in your advice.

You started SymPy and were able to make other people join you and develop it with you.
How did you do it?
How did it happen?
Did you actively call for other people or they spontaneously showed interest and joined you?
Are the other major contributor people who were your friends before you started the project?
Did you need to create or manage the project in a particular way to make it attractive to other people?
Are there things you are aware of that promote collaboration or demote it?

I was never successful in doing the same with Winpdb, which while it became reasonably popular, no one has ever joined me to develop it, except for a notable tutorial contribution by Chris Lasher which was developed independently.

Now with the new project, I am wondering what are my chances of making other people try it and take it on. On the one hand it is a new and fresh code base in an interesting field, on the other hand, why would anyone bother to spend their energy on this new project when they have Symfony or Drupal?

What do you think?

BTW, Ohloh believes you have a median of 19,000 lines of changed code per month since the start of their log. Can this be true? Is this humanly possible? According to it SymPy has over 1,000,000 lines of code? I can't understand these numbers. Winpdb has about 25,000 lines after 3 years of development. And from my experience 1,000,000 lines of code projects need about 20-50 full time developers to work on for 2-5 years which is about 40-250 man years. And as if this is not enough you are listed as owner in a dozen other projects in Google code and have enough time to become an awarded scientist. How is this possible?

http://www.ohloh.net/p/sympy/contributors/

BTW2, do you still use Winpdb? If you find yourself using it less, can you say what are the reasons, or what it would take to make it more useful?

BTW3, How is SymPy doing?

Cheers,
Nir

Are the other major contributor people who were your friends before you started the project?

Did you actively call for other people or they spontaneously showed interest and joined you?

After last week book review on Martin Fowler’s Refactoring, I’d like to review another book, more oriented towards patterns and refactoring.

Content and opinions

First, this book could be seen as a follow up of Refactoring, as almost all the described processes use steps from it. The first chapters explain what to expect from refactoring (how it affect software architecture), patterns and how to detect code that needs refactoring. The catalog is split like the original Design Patterns book in different categories (Creation, Simplification, Generaliation, Protection, Accumulation or Utilities), and each of them is described as in the GoF book (which suits me quite well).

The code language is Java, so it may be sometimes difficult to find the equivalent construction in your favorite language, but it is doable if you known Java basics.

Conclusion

This book quotes several times Martin Fowler, but this is not related to the fact that this book is in the “Fowler signature” collection of the publisher. Refactoring to Patterns addresses what Refactoring couldn’t and goes firther, without imposing, just by suggesting.

Refactoring to Patterns (Addison-Wesley Signature Series) (Hardcover)
by Joshua Kerievsky
ISBN: 0321213351

Price: USD 49.63
52 used & new available from USD 34.98

| 4 | 44

Gary Ruben just asked me if it was possible to retrieve the triangulation information from my previous Delaunay example. Actually the reason I came up with this example is that Emanuelle Gouillart, my partner[*], needed to do Delaunay triangulation on some data. She was kind enough to extract that code from her code base. Here it is.

[*] The various languages do not seem to have evolved quickly enough to cope with the fact that people can now have a stable long-term relationship with someone you are not married to. What word should I be using here: ‘girlfriend’, ‘partner’… ?

I am pleased to announce that EPD (Enthought Python Distribution) version
4.2.30201 has been released.  You may find more information about EPD, as
well as download a 30 day free trial here:

http://www.enthought.com/products/epd.php

You can check out the release notes here:

https://svn.enthought.com/epd/wiki/Py25/4.2.30201/RelNotes

The Enthought Python Distribution (EPD) is a “kitchen-sink-included”
distribution of the Python Programming Language, including over 80
additional tools and libraries. The EPD bundle includes NumPy, SciPy,
IPython, 2D and 3D visualization, database adapters, and a lot of
other tools right out of the box.

http://www.enthought.com/products/epdlibraries.php

It is currently available as a single-click installer for Windows XP (x86),
Mac OS X (a universal binary for OS X 10.4 and above),
RedHat 3, 4 and 5 (x86 and amd64), as well as Solaris 10 (x86).

EPD is free for academic use.  An annual subscription including installation
support is available for individual and commercial use.  Additional
support options, including customization, bug fixes and training classes
are also available:

http://www.enthought.com/products/support_level_table.php

- Ilan

I’ve read this book when I started my PhD thesis. It helped me laying down the basis of software conception.

It was the first book where I found the code smell concept. And my former code really smelt…

Content and opinions

This book became a reference for me. The patterns catalog (because I consider them as patterns) seems almost exhaustive for me. Some of the steps described are too easy (as inlining a function). It may be more difficult at first to extract a method from another, but it is still the basis of refactoring.

The main use of the book isn’t saying what one already know (inling, splitting a function, …), it is showing a new step, one one didn’t think about and that solves the problem at hand. Stating the problem, the smell, is also a main focus of the book, where Martin Fowler gives hints to distinguish between the different smells.

The author is one of the world-known unitary tests herald, so it is no wonder they have a central place in the different patterns. This way, the refactoring cannot change the code behavior. The given examples are also clear and simple enough, completely described step by step (mainly as the book goes forward, new patterns use preceding ones).

Finally, the code is written in Java, but it isn’t a problem for someone used to object-oriented language. Useful tools are also exposed, but they are old, dating from the time when the book was written (and they are mainly geared toward Java).

Conclusion

This was a really simple book. Since I closed it, I didn’t have to open it again, as the patterns are pretty simple. But still, as for every pattern, you first have to understand it (and read it) to acknowledge it. And I tend to refactor regularly following the ideas behind this book.

Refactoring: Improving the Design of Existing Code (Addison-Wesley Object Technology Series) (Hardcover)
by Martin Fowler, Kent Beck, John Brant, William Opdyke, Don Roberts
ISBN: 0201485672

Price: USD 46.79
85 used & new available from USD 30.40

| 4.5 | 138

I was working on a client/server project where we send collections of data across the wire. I needed a method of matching datasets on the client and server, and the python hash function seemed ideal. I suspected that the hash function might have different behaviour on different systems, but conveniently forgot to test it until after I tried to deploy it.

I expected differences, but I didn’t really know to what extent, so I did a little research. So far, ints are the only thing I have found that hash the same, because int’s __hash__ function just returns the int value. Otherwise, Python’s hash functions depend on multiplication using long ints.

While doing my research, I found a page discussing hashing in Python 2.3. The algorithms are similar to the C implementations in Python 2.6.

Of course, I got bit because Python 2.5 on OS X 10.4 and 64bit RedHat 5 didn’t hash my objects the same. In the end, I serialized the data’s metadata and performed a md5 instead, which requres more CPU cycles, but at least it works…

Tonight I sat down and played a bit with VTK’s Delaunay tessalation filter. I wanted to inspect the local structure of a graph created by Delaunay tessalation of random points. To see better the structure, I selected a slab of the resulting unstructured grid. I think the image is not only instructive to explain what a Delaunay tessalation is, it also looks pretty cool. Here is the image and the Mayavi script that creates it.

I missed a date with my wife on Friday to help push the beta release of EPD out for all 10 platforms we are currently supporting (WinXP, WinVista, Mac OS X 10.5(10.4)-intel(ppc), RH3 (x86, amd64), RH5 (x86, amd64)). The 6 different binaries were uploaded to our download servers early Saturday morning (4:00 am Central Time). I’m excited for people to try the new release as it brings together recent NumPy, SciPy, matplotlib, and Ipython together with many additional tools.

One of the things I’m very enthused to have people try is an alpha version of EPDLab which comes in the distribution. EPDLab is an open-source Envisage application which offers an IPython shell along with a linked code editor to allow highly interactive development. EPDLab also contains a “search documentation strings” widget which uses Whoosh and some Robert Kern indexing Fu to provide a very useful search for all of the powerful tools pre-packaged with EPD.

Get the beta2 today and start using a very full-featured distribution of Python across your organization today. Download Beta2.

If you try this recent beta, I’d love to hear from you about any feedback you may have (both positive and negative). Email me at info@enthought.com. The final version of the next release of EPD (4.2.30201) should be out by early next week.

-Travis

If last week’s book review was too complicated for you, perhaps this book is more suited for you. Less design patterns, but a funnier way to describe them.

Content and opinions

Only twelve patterns are explained, but more important is the fact that each of them is detailled, with examples and exercices, as well as important phrases displayed as images. With humor, these are the caracteristics of the “Head First” collection.

Instead of using the same example throughout the book, each chapter has more or less its own concrete example. The latest chapters explain how they can interact to create complex applications, or how they can define new design patterns (as MVC for instance), then how it will change your way of thinking software architecture.

Conclusion

This book focuses on the most used and perhaps useful design patterns. This way, it can present them differently than a simple catalog, but I have to say that this approach will not be suited for everyone. For instance, I appreciate books that go straight to the point.

This kind of book is well suited for people that want to start with design patterns, but not for people familiar with them: they should go for catalogs.

Head First Design Patterns (Paperback)
by Elisabeth Freeman, Eric Freeman, Bert Bates, Kathy Sierra
ISBN: 0596007124

Price: USD 29.67
77 used & new available from USD 19.94

| 4.5 | 273

As I’ve said before, I’ve done several book reviews in the past. I will start with a small serie on design patterns books.

This book is one of the “must-have” in your library. If you write some code or if you manage some IT or Computer Science projects, you will have this book to lay down the basic software architecture.

Content and opinions

The first two chapters are an introduction and explain the reasons of the existence of design patterns, how they should be used, good and bad pratices, … Design patterns without rules to apply them are useless (as the original architectural patterns are useless without drawings skills). A practical example is the object of the second chapter.

Design patterns are exposed in a three parts catalog. Each pattern each described by a complete explanation, an UML diagram, the interactions between the pattern elements, as well as some implementation solutions (all solutions cannot be written, as it is language-dependent).
Creational patterns are about creating new objects. They include the abstract factory (constructing several objects of different kinds), the builder (a more elaborate constructor), the factory method (overloading a class method to create objects based on different classes), the prototype (creating new objects by cloning an instance) and the singleton (creating only one instance of a given class).
Structural patterns are more about the actual software architecture. They include the adapter (translating an interface to another one), the bridge (separating an interface from different implementations), the composite (allowing several objects of a hierarchy to be composed together), the decorator (adding characteristics to an object), the facade (offering an interface to several classes), the flyweight (sharing the same objects between instances so as to reduce memory overhead) or the proxy (using another object to access another, potentially hidden, one).
Behavioral patterns enable a software to change its own behavior. They include the chain of responsibility (allowing processing requests by whoever can), the command (creating complex requests), the interpreter (describing how a language can be processed), the iterator (providing a way of browing the content of a data container), the mediator (allowing communication between different classes), the memento (enabling restoring the state of an object), the observer (sometimes also called listener, creating a way for instances to be updated/called by another one), the state (allowing changing behavior on the fly), the strategy (providing several ways of doing something), the template method (providing a skeletton for an algorithm) or the visitor (allowing execution of code for every the content of an object). This patterns set is perhaps the most heterogeneous one (although the state and the strategy are in fact exactly the same, the only difference being the interpretation of their actions).

Conclusion

This book, sometimes refered ad the GoF book, lays down the basis of software conception. These 23 patterns are not the only ones you may use (some of them are also seldom used), but they are used to make the more complicated ones. If you have one design patterns book you should buy, it is this one.

Design Patterns: Elements of Reusable Object-Oriented Software (Addison-Wesley Professional Computing Series) (Hardcover)
by Erich Gamma, Richard Helm, Ralph Johnson, John M. Vlissides
ISBN: 0201633612

Price: USD 38.96
124 used & new available from USD 23.06

| 4.5 | 261

We’ve had a number of recent internal discussions about EPD during which the phrases “that won’t work on OS X 10.4″ or “does upstream have PPC support?” came up quite often.  For example, a recent discussion about the importance of relocatable EPD egg installs sputtered because we realized Mac OS X 10.4 doesn’t support RPATH settings in binary headers, which meant we’d have to do something special just for that platform.

Once we realized this commonality, we next wondered how important OS X 10.4 and PPC support actually is for the EPD user community.  Thus the point of this blog post: to get some community input.  This is your chance to speak up if you need OS X 10.4 and/or PPC support.  I can’t promise that a single ‘yes’ will sway our decision making, but certainly the more people who speak up, the more likely we are to try to continue the support.

For those who don’t know, sys.path is the path that the Python interpreter traverse at each module import to look for the module file imported.

This blog post is about the consequences of having a long sys.path. I’ll try and make it short, but I would have a lot to say. I am just reacting on Noah Gift’s post on performance improvement, not making a full essay on why overloading sys.path is considered harmful.

When using easy_install (or setuptools), each new project is installed in a different directory, and the directory is added at runtime to the sys.path (the addition at runtime confuses many users who are not aware of it). As a result, you quickly end up with more than 40 directory on your sys.path. These directories are ’stat-ed’ one after the other on each module import. Thus if you have a long sys.path, there are a large amount of system calls to read directories. To check this out, simply try:

strace python -c "import foobar" 2>&1 | less

You can see the amount of noise created by a simple (failing) import statement. On a system with high latency (such as an NFS, as we use at work), this is very costly.

Noah joyfully reports performance improvements by hijacking the Python import mechanism. I claim that part of what Noah has done is not really hijacking the import mechanism, it is undoing the hijacking performed by setuptools.

I know I am being rude, but many people raised this point before, and it is not getting any traction from the setuptools maintainer. I claim that you should not be using setuptools or easy_install if you want performance or control. I claim that you should not be using setuptools unless you understand well what you are doing (which defeats the name easy_install).

The way I install packages when I want good control via easy_install is in a virtual environment to discovered the dependencies, and then:

easy_install -Zeab . package_name

to download the package for each required package, and

python setup.py install --single-version-externally-managed –record ./foobar

if the package itself is using setuptools.

As you can see, setuptools make it really hard to do a clean install. Its a design choice :(.

Another alternative is to use pip which I strongly encourage.

Enthought is offering “Introduction to Scientific Computing in Python” at our offices in Austin, Texas from June 15th to June 19th. This course is intended for scientists and engineers who want to learn to use Python for day-to-day computational tasks.

• Day 1: Introduction to the Python Language
• Day 2: Array Calculations with NumPy
• Day 3: Numeric Algorithms with SciPy
• Day 4: Interfacing Python with Other Languages
• Day 5: Interactive 2D Visualization with Chaco

The cost for the course is $2500. Please see the course description on the Enthought website for details.

Space is still available in our course on Python for Science, Engineering, and Financial Analysis, May 18th to 21st, in New York City

Profiling comes in three different flaviors. The first is emulation, where a processor behavior is emulated, the second is sampling, where at regular intervals, the profiler samples the status of a program, and fianlly instrulentation, where the profiler gets information when a subroutine is called and when it returns. As with the Heisenberg uncertainty, profiling changes the exact behavior of your program. This is something you have to remember when analyzing a profile.

Valgrind is an Open Source emulation profiler. It is freely available on standard Linux platforms. As it is an emulation, it is far slower than the actual program. This means that the I/O are underestimated. The advantage is that you can have every detail on the memory behavior (cache misses for instance). Valgrind does not emulate all processors, but you can tweak it to approach your own one.

This is more or less a translation of my French tutorial on Valgrind profiling.

Calling Valgrind

Calling the profiler is really easy:

valgrind --tool=callgrind --dump-instr=yes --simulate-cache=yes --collect-jumps=yes program arguments

Here, I ask valgrind to use the callgrind profiler plugin, and it is supposed to dump the executed instructions (which will help knowing which part of a function really costs, not only which function), simulate the cache (to help enhancing the processor usage) and collect jumps (to have a dynamic view of the program behavior). Of course, the program must have been compiled with the appropriate compilation options (at least -g).

Analyzing profiles

KCacheGrind is probably the best tool to visualize ad analyze valgrind results (it can also display other profilers results).

When opening a profile, KCacheGrind may not recognize the associated source files. You may add their folder to the annotations folders.

I think the most important graph KCacheGrind provides is the Callee Map. It can be colorized by different means (files, classes, …), the main point being that Callee Map provides an image where the surface of a function represents its weight in the program execution (weight being number of instructions, cache misses, …). Unfortunately it appears that in some cases, KCacheGrind is not able to create everything Callee related. I don’t know why, but I got this on a RedHat 4, the associate KCacheGrind and the latest valgrind.

Call graphs can also provide intel on how much each function consumes. When double-clicking on a function (in the call graph, in the Callee Map), it is “activated”. The original source code is shown (with jumps, if they were collected) with the cost for each instruction, which functions called the function and which functions are called. Another important thing is the difference between the self cost (sometimes called exclusive cost) and the inclusive cost. The former is the cost of the function alone, the latter is the cost of the function with the cost of the called functions.

Conclusion

Valgrind combined with KCacheGrind are free tools to make an application profile. It is far from perfect, but it provides valuable information. Instrumentation- and sample-based profiles need a patched kernel (for Linux) or administrator rights (for Windows and Linux), and they can’t provide at the moment every cost, contrary to emulation.

• robust and fast (still pure Python) multivariate factorization
• sympy works with pickle protocol 2 (thus works in ipython parallel)
• ./sympy test now uses our testing suite and it tests both regular tests and doctests
• examples directory tidied up
• more trigonometric simplifications
• polynomial roots finding and factoring vastly improved
• mpmath updated
• many bugfixes (more than 200 patches since the last release)

• Ondrej Certik
• Mateusz Paprocki
• Fabian Seoane
• Andy R. Terrel
• Freddie Witherden
• Robert Kern
• Priit Laes
• Riccardo Gori
• Fredrik Johansson
• Aaron Meurer
• Alan Bromborsky
• Brian E. Granger
• Felix Kaiser
• Kirill Smelkov
• Vinzent Steinberg
• Akshay Srinivasan
• Andrew Docherty
• Andrew Straw
• Henrik Johansson
• Kaifeng Zhu
• Ted Horst

• Riccardo Gori
• Fabian Seoane
• Vinzent Steinberg
• Gael Varoquaux
• Fredrik Johansson
• Robert Kern
• Alan Bromborsky
• Ondrej Certik