Saturday, March 13, 2010

Comparing the Ruby/PHP/Python C Interpreters

The other day I went poking around the Ruby and PHP interpreters (the current stable versions). I hadn't looked inside PHP since the 4.x series and Ruby I had never checked out. Like CPython the internals of both PHP and Ruby look something like their resulting language, but in C. For each interpreter I just compiled it and looked at how core types and extension types were implemented.

Ruby 1.9.1

Ruby-the-language has lots of syntax and its core types are just as extensible at run time as classes written in ruby (you can monkey patch core types). The compile was clean and runs with -Wall, generating just a couple warnings. All the unit tests passed. The grammar is implemented with lex/yacc and the resulting parse.c file took 10 minutes to compile on my 1.5GHz machine. Did I mention the grammar is big?

There is no difference between ruby core types and extension types written in C. That is mostly true in python but ruby goes all the way. The C-struct that holds information about the ruby type has a hash map that contains all the type's methods - and I mean all of them. Here is the interface for adding a __add__ method (cBignum is the core integer type)

rb_define_method(rb_cBignum, "+", rb_big_plus, 1);

The "+" is the not-so-magic name for the addition operator. The type's hash uses "+" as a key that points to the value of the addition function. That is a beautiful interface compared to CPython, where you have to put the __add__ method in the right place in a struct[1]. As an optimization the "hash" is actually a list if the number of methods is small; method strings are interned and assigned a number - I'm not sure why this is faster than just keeping the hashkey on the string and always using a dict, but I assume someone benchmarked it.

PHP 5.2.13

[NB, I should have looked at the 5.3.x release but the 5.2.13 release was at the top of the homepage when I went looking]

I hadn't looked at PHP since the 4.x series (see my why I started using Python post). PHP has added some nice features since then, like namespaces, but the interpreter looks much the same. The compile uses a custom wrapper around gcc and is very spammy: a dozen -I include directories on each line for hundreds of C files. It does not use -Wall by default so if you want really really spammy turn that on. After compiling PHP I ran the unit tests and 7 failed[2]. All 7 had to do with bad conversions between signed and unsigned numbers (a negative signed int is a positive unsigned int). This is a production release so those failures are not confidence inspiring.

Like PHP-the-language the C interpreter makes a big distinction between core types and extension types. The core types are int, string, and list/hash (a hybrid). The C-struct is a union that has is either an integer, string, list/hash, or "resource" (everything else). Extension types can't do operator overloading so the interpreter has if/else clauses for handling the core types. Methods are added by registering them by resource number in a global registry.

Objects get passed around in the core as pointers to pointers, and sometimes as pointers to pointers to pointers. I'm not sure why, but this can't be good for speed.

Python 2.5+ 3.x

I'll lump all releases of Python after 2.5 together because the internals are very similar. The AST (abstract syntax tree) that the byte compiler uses was rewritten and simplified for the 2.5 release and there haven't been any big changes to the internals since then. The 3.x releases made some big simplifications, but they still use the same framework.

Like Ruby, Python compiles cleanly and uses -Wall, generating few warnings. The test suite passes. Python doesn't make a distinction between core types and extension types: if you copied Objects/dictobject.c and renamed it "mydict" [insert dict joke here] you could ship it as a module and "import mydict". The only difference is that the byte compiler knows that when you type "d = {}" you mean "d = dict()".

The C-struct for python types is a bit more complicated than the ruby one. It has specific slots for all the magic methods like __add__ instead of keeping them in a hash map like it does for pure-python classes. Like PHP the execution loop does have some if/elses for core types like integer, but unlike PHP this is just a speed hack and not a requirement (I assume Ruby does something similar).


So there you have it. All three interpreters look much like their parent language once you get under the hood. I'd mention the perl interpreter too but it's been years since I dove into that one; but guess what? It looks like perl.

[1] python-dev has several threads about adding a similar simple interface. Someone just has to do the work (at PyCon Hastings said he's exploring it).
[2] I downloaded PHP 5.3.2 and the 7 test failures I saw are fixed, but I get 9 new and different failures.

PS, blogger hates H4 tags. Why the extra newline?


cotery said...

So PHP internals are a crappy mess. Wotta surprise.

zenspider said...

Huh? The grammar has never been so complicated that it took 10 minutes to compile. On my 2.13 Ghz macbook air ruby 1.8 takes 10 seconds to generate and compile parse.c and ruby 1.9 takes 11 seconds to parse and generate.

Mix up your units maybe?

Marius Gedminas said...

The pointer-to-pointer thing: maybe it's there to support compacting GC? Then a pointer-to-pointer-to-pointer thing could be an output parameter.

Just guessing; I'm clueless about PHP.

Jack Diederich said...

@zenspider gcc 4.3.3 on an AMD Sempron 2800, ruby-1.9.1-p376

$time make parse.o
real 11m46.144s
user 9m56.777s
sys 0m6.776s

Jack Diederich said...

Someone started a reddit comment thread

tenderlove said...

@Jack Diederich

That's crazy! I'm working from svn trunk, so maybe it's different, but:

$ gcc -v
Using built-in specs.
Target: i686-apple-darwin10
Configured with: /var/tmp/gcc/gcc-5646.1~2/src/configure --disable-checking --enable-werror --prefix=/usr --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin10 --with-gxx-include-dir=/include/c++/4.2.1 --program-prefix=i686-apple-darwin10- --host=x86_64-apple-darwin10 --target=i686-apple-darwin10
Thread model: posix
gcc version 4.2.1 (Apple Inc. build 5646) (dot 1)
$ time make parse.o
gcc -O3 -ggdb -Wextra -Wno-unused-parameter -Wno-parentheses -Wpointer-arith -Wwrite-strings -Wno-missing-field-initializers -Wshorten-64-to-32 -Wno-long-long -pipe -I. -I.ext/include/x86_64-darwin10.2.0 -I./include -I. -DRUBY_EXPORT -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -o parse.o -c parse.c

real 0m9.967s
user 0m9.547s
sys 0m0.414s

postmodern said...

I concur with @zenspider and @tenderlove. Compiling a clean parse.o for Ruby 1.9.1-p376 on my AMD Athlon 64 3400+ takes less than a minute:

make clean; time make parse.o
gcc -O2 -g -Wall -Wno-parentheses -fPIC -I. -I.ext/include/x86_64-linux -I./include -I. -DRUBY_EXPORT -o parse.o -c parse.c

real 0m10.894s
user 0m10.380s
sys 0m0.360s

Jack Diederich said...

I guess gcc 4.3 has some more CPU-intensive optimizations than 4.2. The parse.c file is 16KLOCs which includes machine generated code; so something must be sending gcc -O2 into "deep think" mode.

Also, I went back and checked and I was slightly off in my description of how Ruby deals with small method lists. It just scans the list linearly and checks the keys for equality. Basically like a hash table with 100% hash collisions. I presume this is to save a little memory and maybe to avoid hashing (they are interned strings so they are probably pre-hashed).

I would update the original post but then it would get bumped to the top of planet.python, which is annoying.

taama said...

.8 takes 10 seconds to generate and compile cape town rentals parse.c and ruby 1.9 takes 11 seconds to parse and generate.

php code generator said...

I swear, I have to leave my company, we do critical work on 4 year old machines, I dream of a better programming future :(