Jack Diederich's Python Blog

PyCooooooonnnnn!

2011-12-23T11:05:00.003-05:00

The Talk schedule has been published and I'm on it. For which I'm very greatful - I saw the full list of ~400 submissions and there were many I'd like to see which didn't make the cut. [To all those people I'll let you in on a secret: submit the same thing at a smaller conference. They have 80%+ acceptance rates - basically they just toss out the crazies and let everyone else speak. PyCon was like that not too long ago. So go and speak wherever they will have you and get some practice: my first PyCon talk never would have been accepted under today's standards.]

Name Dropping

Normally I'm a Hallway Track kind of guy but there are some talks from friends I kinda want to see as it happens as an event: Larry Hastings, Doug Hellman, Jeff Rush, Benjamin Peterson (is he old enough to drink yet?), David Mertz, Moshe Zadka (long time/first time?), Ned Batchelder, Alex Martelli, Barry Warsaw, Dino Viehland, Maciej/Gaynor/Rigo (pypy), Armin Ronacher, Glyph (I'll tell you his real name in trade for a beer), Raymond Hettinger, Gary Bernhardt, and Jim Baker.

And that's just the list of speakers I know will be good, there are many more that look good on paper that I want to see.

You'd better nail it kid

There are some perennial talk topics because it is always nice to have a fresh take on important things. That said, being the second (or third, or fourth) person to retread the same ground is difficult.

Larry Hastings is treading old ground with Stepping through CPython, a tour of what really happens when your python code runs. I did it in 2006 as "Writing your own python types in C" and Ned Batchelder did it much better in 2009 as "A Whirlwind Excursion Through Python in C". I'm looking forward to Larry's take.

Getting the most out of Python imports by Eric Snow. Note that he speaking about imports and his name isn't Brett Cannon.

The Magic of Metaprogramming by Jeff Rush and Metaclasses by Luke Sneeringer. Not just well tread ground, but ground always loaded with fresh landmines.

Decorators and Context Managers by Dave Brondsema. I'm very much looking forward to this one, as those things have been my topics in the last few years. I haven't met Dave but I imagine he has giant brass balls.

And the other thing

Raymond Hettinger will be presenting The Art of Subclassing and I will be presenting Stop Writing Classes. Raymond and I have agreed on pretty much everything in the decade I've been involved with python (he was already a commiter when I showed up*). I doubt we are in disagreement about classes or subclassing, rather we are giving talks about two different topics. His is "here is how you do hard things well" and mine is "don't do hard things, they are hard and you'll probably fuck it up". Two weeks ago I wrote a caching class that used __new__ and this past week I refactored it to not do that; the code was half the length and achieved the same result. My only question to the team was "why did no one yell at me two weeks ago?"

* My favorite PyCon story is my first one: 2003 in DC and I got to meet everyone that was previously just names on a mailing list. Hettinger was working the registration desk and I was like "thats neat" and when I stepped out into the smoker's lounge I introduced myself to the three hangers about. "Hi, I'm Jack Diederich" was followed by "Hi, I'm (Alex Martelli|Tim Peters|Christian Tismer)". That was awesome.

Analog Analogies

2011-07-11T23:54:00.004-04:00

At work there has been a recent interest in explaining tech things in terms non-tech people can understand. Not just because it helps explain tech things to non-tech people but because it also clarifies to tech people what the purpose of the thing is.

We hit on the term "analog analogies" because one of my coworkers kept saying "analog" instead of "analogy" and I didn't think of the two terms as synonymous. I thought of "analog" as opposite "digital", but not for a second did I think "analogue" was synonymous with "analog" (it is). Hence the redundancy. So "analog analogies" became the phrase meaning "explaining a digital thing by analogy."

By way of for example[1] I'll be leading off my five minutes at the Q3 kickoff with one of my favorite analog analogies: the high and low laundry equilibriums. There are only two stable equalibriums in laundry: you can keep all your clothes clean and do laundry every time you have a basket of dirty, or you can have a full clothes hamper and only do a load when you need something to wear.

It is a not terrible analogy for technical debt. But the analogy falls down because even in a medium sized project you might have several different laundry equilibriums across different aspects: are we keeping a lid on our code bottlenecks? backend queries? can we rent/buy faster machines? And the worst part is you might not know which is lacking. I think the Germans have a word for how you know: messerschitz. You gotta do that too.

[1] I like to pretend to be all Dutchy on occasion.

MUD Nostalgia

2011-06-05T21:27:00.002-04:00

For programmers of a certain age - those who saw their first internet connection in the 90s - MUDs [Multi User Dungeons - basically D&D online] were de rigueur. Open Source was in its infancy and this new thing called linux came on a stack of 35 3.5" floppy disks (configuration was equally hostile). So if you wanted to contribute to a collaborative project MUDs were where it was at. A number of python people have outed themselves as [ex?]mudders, including Glyph (Twisted founder, Twisted started as a mudlib), Doug Napoleone (PSF board), Jesse Noller (PSF board), Richard Tew (Stackless Python), Will Kahn-Greene (Miro, and mud client Lyntin), and yours truly (I maintain telnetlib, need I say more?).

Learning is in the doing, so what I got of college was mostly lots of free time to do stuff. I spent maybe 400 hours in four years doing programming for classwork and over 4000 hours writing LPC code for a now defunt MUD* (using /bin/ed - the only supported editor). Writing a mudlib was my "turtles all the way down" moment, when it clicked that user level programs were just programs that were composited from other programs; The trick was that the lower level stuff had an agreed upon naming convention but otherwise wasn't special at all.

I'll admit to being somewhat jealous of the college students of today - it is easy to contribute to projects that actually matter. Being able to rely on ping times being under 500ms is nice too (Europe used to just _go away_, sometimes for days).

I haven't left MUDing completely behind. I still use it to try out new methodologies in coding and just play around in general. Leanlyn (http://bit.ly/leanlyn) is my Lyntin fork that is about 10 years old and will never be finished (or be less than user hostile).

* The dead mud was Ether Realms. I can still be found sometimes at OverDrive (a Lehigh staple) and Three Kingdoms.
** College came with lots and lots of free time. I spent way more than 4000 hours just chasing tail. Imagine me but with shoulder length hair, an eyebrow ring, and a flannel shirt (it was the 90s)

PyCon Speaking

2011-03-16T21:33:00.005-04:00

Do read Moshe's How to give an OK talk, esp the Narrative part. It feels quite silly to repeat yourself on every slide but you are preforming live and not writing on paper - so tell the audience why slide N+1 is important and for the same reason as slide N at every opportunity. During my talk I felt as though I was being overly repetitive, but on watching it after the fact I feel like I didn't hammer home the point enough (you should do more shit like this!).

Speaking of speaking: I asked a few popular conference warriors (Martelli & Hettinger) how many talks they had given and when they first felt comfy giving talks. They didn't know. I should have asked Beazely and C.T. Brown but they're professors and get more practice in a year than most people get in a lifetime.

But seriously, those popular conference regulars didn't know. Hettinger estimated he had spoken 30 times. His wife then pointed out he did 15 talks last year and so he upped his estimate to "more than 15 twice." Martelli estimated 50-100 talks but was equally uncertain. Myself, I've only done 7-8 and still get nervous if the room is less than 30 or more than 300 people. Hettinger said something like "I still get nervy when it the audience is 10x what I'm used to." I couldn't coax an admission out of Alex (I had only vague thoughts on the topic when I asked, and no narrative).

Take it all as paraphrased hearsay, but anytime I have to step up to a podium I just say to myself "suck it up, princess." It mostly works.

PyCon Detritus

2011-03-12T17:47:00.008-05:00

PyCon proper is over and sprints have begun.

Language Summit

The Summit was boring again this year, and this is good. The language moratorium is over so PEP 380 ("yield from") is moving forward. The stdlib will get its own repository so it can be shared between CPython/Jython/IronPython/PyPy. Likewise all the commiters from the not-CPython projects will have commit privs to CPython (most already do).

My Talk

Useful Namespaces: Decorators and Context Managers went over well: video available here and here's the live Convore thread from the talk.

Every year I say I'm not going to do another talk, but it is years like this that make it worthwhile. Dozens of people I don't know have offered me thanks, jobs, free beer, and ZJs. There is no worse feeling than spending 40 hours on something and having it be less than good. I'm a happy camper.

The first question was asked by Larry Hastings ; it's become an inside joke because Larry has asked the first question at ALL my past PyCon talks (having a confederate to break the ice on questions is handy). I zinged him a bit in my response ("thank you for pointing that out, and may I point out it's just an example slide"); apparently most people thought I was razzing a random questioner even though I was addressing him by name.

Another questioner objected to my use of a module named "UnicodeNazi" because it isn't nice to talk about Nazis even (or especially) jokingly. I restated his objection and said he should take it up with the module's maintainer. That maintainer is Austrian, natch.

The room was pretty full. Not going out the night before definitely helped my delivery. There were some A/V problems at the start (lots of those this year) so I rushed the last few slides. Better that than running short though.

Other People's Talks

So far, so good. Yesterday my talk was in the afternoon so I spent all day preparing and didn't make any talks. Today I went to three: Alex Gaynor's "Python's Data Structures," Larry Hasting's "The Python That Wasn't," and Richard Saunder's "Everything You Wanted to Know About Pickling..." Most of my time was consumed by the Hallway track.

Hotel

The hotel has banned alcohol (or at least implied a ban) in public spaces that is not purchased from the hotel. I don't know if this is because of money or just because they want fewer drunks walking around. Annoying either way.

Diverstiry

A woman came up to me and said that something I told her last year had stuck with her. Last year she was sitting at a table with myself and another core dev, and holding her own when discussing eldrich corners of Python. The topic turned to contributing to open source and she had a list of reasons she felt she couldn't contribute, none of them very good. The good news is that she sprinted remotely last year, is sprinting in person this year, and has contributed code to projects over the past year. Oh, and the thing I said that stuck with her? "Suck it up, princess." So I have struck a blow both for and against diversity. [FYI, I don't remember saying that but it is the kind of thing I might say after a few beers]

Happy PyCon to Everyone

2011-02-26T18:08:00.003-05:00

I'm in the happy majority of Python commiters that mostly idle except for the scheduled holidays like bug weekends and release alphas, and only awake for the high holidays like PyCon. I'll be speaking again this year and I'll be there for the language summit before and two days of sprinting afterward.

This weekend I stopped researching my talk "Useful Namespaces" and started actually writing slides; the theme is a riff on "Namespaces are one honking great idea -- let's do more of those!" and the first slide is "I monkey patch the shit out of everything." Coincidentally I won't be sprinting 100% on core stdlib this year but will instead be trying to fix up my monkey patches and tests on other projects that haven't yet made it upstream.

On Deployment Policy

2010-09-24T16:48:00.002-04:00

Q: Don't we have a policy about drinking during deployments?
A: Yes, we always drink during deployments.
Q: I thought the policy was "Friends don't let friends deploy drunk?"
A: If you can still type how drunk can you be?

Encouraging Arbitrage in Virtual Economies

2010-04-08T12:48:00.002-04:00

I'm always surprised by how different companies conduct interviews - I shouldn't be after being surprised so many times; so my surprise at being surprised is surprising (to be all meta). Today I had to give a talk to a group of devs on the topic of Anything. I started out with snippets of my canned talks and then it diverged as people asked questions. Anyway, it is a virtual goods company so here is the talk I should have given, but the idea didn't hit me until I was in the car on the way home.

Encouraging Arbitrage in Virtual Economies

Wikipedia defines arbitrage as

In economics and finance, arbitrage is the practice of taking advantage of a price difference between two or more markets: striking a combination of matching deals that capitalize upon the imbalance, the profit being the difference between the market prices.

So arbitrage is being the classic and much maligned middle-man: buy from one guy and then resell to another guy at a slightly higher price. People hate middle-men because they seem to make money from doing nothing (they don't make anything dammit!), but they serve the very useful function of smoothing out prices. In return for their cut they add information to the system and more information is a good thing.

When there are a lot of people arbitraging a good you get more stables prices. When there are several pawnshops close by you can bet that they will charge very similar prices. You can walk into one and sell your iPod without doing research and be pretty sure you got a fair price.

When there is competition the gap between what you get paid for that iPod and the price of buying one also shrinks. This has huge ramifications because it basically allows you to rent items by buying them and then selling them back. Need an ebook reader for the weekend? buy it today and sell it back on Monday for $10 less.

What this means for virtual economies

In a heavily arbitraged market users can try out new things cheaply: instead of renting that ebook reader they can try out a FarmVille tractor or that fancy new Vorpal blade, safe in the knowledge that they can sell it back at a small loss. Users like to do new things, so this is good. Since arbitrage adds information to the system it also tells users what they should be doing. They can see through prices what other users value and then do more of that.

Note that I added some big qualifiers in there: heavily arbitraged, competition, a lot of people. To get those things you need to make arbitrage as easy as possible. You need a market that supports both "bid" and "ask" prices so the gap is obvious even to casual users. You also want as many people doing hard-core arbitrage as possible so give them as much information on past prices as possible in a very easy to use format. I really can't emphasize that enough: I've ruined many in-game economies by actively removing information from the system by say, buying up all the copies of an item. Other people didn't have any current information on the prices, and the games didn't offer historical information so it was only me who had it. This is possible even on sites that offer some historical information (Duels only listed the last N sales so I just sold to myself a bunch of times to clear the buffer).

If you have a bunch of people doing this and the transactions only happen in-game you also have a ready made money sink. Tax the transactions at X% and put the proceeds in /dev/null. An interesting question is what that tax rate should be. If it is high you get less arbitrage because there is no money to be made, and as a knock on the other users get less benefit because the cost of "renting" an item is higher.

PS, on a python-related note: there are lots of python folks that care about this stuff in real life markets too. Corner Hettinger or Glyph sometime when you have a few hours to spare.

Comparing the Ruby/PHP/Python C Interpreters

2010-03-13T09:26:00.002-05:00

The other day I went poking around the Ruby and PHP interpreters (the current stable versions). I hadn't looked inside PHP since the 4.x series and Ruby I had never checked out. Like CPython the internals of both PHP and Ruby look something like their resulting language, but in C. For each interpreter I just compiled it and looked at how core types and extension types were implemented.

Ruby 1.9.1

Ruby-the-language has lots of syntax and its core types are just as extensible at run time as classes written in ruby (you can monkey patch core types). The compile was clean and runs with -Wall, generating just a couple warnings. All the unit tests passed. The grammar is implemented with lex/yacc and the resulting parse.c file took 10 minutes to compile on my 1.5GHz machine. Did I mention the grammar is big?

There is no difference between ruby core types and extension types written in C. That is mostly true in python but ruby goes all the way. The C-struct that holds information about the ruby type has a hash map that contains all the type's methods - and I mean all of them. Here is the interface for adding a __add__ method (cBignum is the core integer type)

rb_define_method(rb_cBignum, "+", rb_big_plus, 1);

The "+" is the not-so-magic name for the addition operator. The type's hash uses "+" as a key that points to the value of the addition function. That is a beautiful interface compared to CPython, where you have to put the __add__ method in the right place in a struct[1]. As an optimization the "hash" is actually a list if the number of methods is small; method strings are interned and assigned a number - I'm not sure why this is faster than just keeping the hashkey on the string and always using a dict, but I assume someone benchmarked it.

PHP 5.2.13

[NB, I should have looked at the 5.3.x release but the 5.2.13 release was at the top of the homepage when I went looking]

I hadn't looked at PHP since the 4.x series (see my why I started using Python post). PHP has added some nice features since then, like namespaces, but the interpreter looks much the same. The compile uses a custom wrapper around gcc and is very spammy: a dozen -I include directories on each line for hundreds of C files. It does not use -Wall by default so if you want really really spammy turn that on. After compiling PHP I ran the unit tests and 7 failed[2]. All 7 had to do with bad conversions between signed and unsigned numbers (a negative signed int is a positive unsigned int). This is a production release so those failures are not confidence inspiring.

Like PHP-the-language the C interpreter makes a big distinction between core types and extension types. The core types are int, string, and list/hash (a hybrid). The C-struct is a union that has is either an integer, string, list/hash, or "resource" (everything else). Extension types can't do operator overloading so the interpreter has if/else clauses for handling the core types. Methods are added by registering them by resource number in a global registry.

Objects get passed around in the core as pointers to pointers, and sometimes as pointers to pointers to pointers. I'm not sure why, but this can't be good for speed.

Python 2.5+ 3.x

I'll lump all releases of Python after 2.5 together because the internals are very similar. The AST (abstract syntax tree) that the byte compiler uses was rewritten and simplified for the 2.5 release and there haven't been any big changes to the internals since then. The 3.x releases made some big simplifications, but they still use the same framework.

Like Ruby, Python compiles cleanly and uses -Wall, generating few warnings. The test suite passes. Python doesn't make a distinction between core types and extension types: if you copied Objects/dictobject.c and renamed it "mydict" [insert dict joke here] you could ship it as a module and "import mydict". The only difference is that the byte compiler knows that when you type "d = {}" you mean "d = dict()".

The C-struct for python types is a bit more complicated than the ruby one. It has specific slots for all the magic methods like __add__ instead of keeping them in a hash map like it does for pure-python classes. Like PHP the execution loop does have some if/elses for core types like integer, but unlike PHP this is just a speed hack and not a requirement (I assume Ruby does something similar).

Conclusion

So there you have it. All three interpreters look much like their parent language once you get under the hood. I'd mention the perl interpreter too but it's been years since I dove into that one; but guess what? It looks like perl.

[1] python-dev has several threads about adding a similar simple interface. Someone just has to do the work (at PyCon Hastings said he's exploring it).
[2] I downloaded PHP 5.3.2 and the 7 test failures I saw are fixed, but I get 9 new and different failures.

PS, blogger hates H4 tags. Why the extra newline?

Some Odd Observations as a PyCon Speaker

2010-03-05T23:22:00.002-05:00

1) Your answer to the first question after your talk will be simple, neat, and wrong.
2) That question will have been asked by Larry Hastings.

Extrapolation from my experience might fail in your particular circumstances because Larry isn't omnipresent. It might fail for my talks too: the pycon video archives only go back to last year.

NB "wrong" in the sense of "less than optimally correct." I included a fuller answer on my published slides both years. Which no one will see.

PyCon Wrapup II: Python Stuff

2010-03-02T00:27:00.009-05:00

[People stuff is trees-and-forest, so here is a post on what was done about Python at PyCon]

The talks were good and the 5x (as opposed to 4x) tracks didn't seem to hurt. Worst case: any talk you missed you can watch on pycon.blip.tv. Speakers were aware that their talks would be recorded so using laser pointers (instead of highlights or spoken words) is going away. This is a sideways move - laser pointers were useful right up until they weren't.

The Language Summit was far more boring than last year. Python 3.x issues are mostly settled from the core-dev standpoint so the big issues were disutils (how, and at what level should python packages care about packaging) and alternate implementations. Unladen Swallow was the talk of the town not because they are the first alternate implementation of Python but because they are the first implementation that plans to ship with benefits and no tradeoffs. Did I say no tradeoffs? It was unanimus that both disutils2 and Unladen Swallow would be integrated once the tradeoffs were wholly positive. Who can't get behind that?

PyCon sprints were smaller for python-dev this year but I couldn't tell if that was true for other major groups like Twisted and Django (the rooms were more broken up this year). I can say for certain the python-dev sprint had both fewer regulars and fewer pure-newbies; One local EE postgrad wanted to help out and simultaneously tried to make me care that his badge and his name didn't match; He went by "Cedric" but his badge said something else (I believe he was Caribbean). I have a long standing amusement with first names: my birth certificate doesn't say "Jack," Titus Brown's doesn't start with "Titus," and Alex Martelli's doesn't say "Alex." For that matter Guido's may say "Guido" but he doesn't care how you pronounce it. In fact all the groups I'm a participant and care about most don't care who you are legally, and don't ask for ID at the door (to riff on my last post, caring about legal ID is a "negative trust" cue).

The PSF (Python Software Foundation) exists to serve two different classes of users: end-users and People-who-hire-end-users. To put it differently the PSF is a single purpose organization that wants more users both from the bottom-up and top-down. The bottom-up stuff has been easier to organize in the form of "your-locality-here" Cons. While the PSF wants to help people to do more of that they also want to aid the corporate users who have an interest in Python. Getting companies to spend money and organize sprints has happened quite successfully before, but very irregularly (see the Need for Speed sprint). Quite happily I can say that if half the events that were spit-balled at PyCon come to be then sprints will be even more prolific in the near future (both bottom-up and top-down) and they will be just as free but even more topic specific (2to3 porting, hardcore dev stuff). At least four groups have intentions to do an event in Boston, for instance.

The CPython bug tracker has 2000 outstanding issues (a mix of bug reports/feature requests/doc requests). A new status field named "languishing" was added because there are a lot of bugs that have +1/-1 comments by core-devs but no resolution; it is a classic "middle school dance"* deadlock where no-one feels they have authority and is just waiting for someone else to pronounce. AMK and I closed about 20 of these during sprints (some applied, some rejected) but there are still a ton of these bugs outstanding. They just need a champion (for or against) to get resolution. Alex Gaynor recently did a post about who-gets-what-commit-rights on various python projects (not including python-dev). Python-dev can be disfunctional because there are 120 committers and everyone assumes there is someone else who knows better for any particular bug (the "middle school dance").

Steve Holden and Michael Foord both have the audio bug: imagine what camera crazy people spend on digital SLR gear but apply that to audio eq. Between the two they captured tens of hours of audio at PyCon and some of that should start showing up soon. Editing is the hard part in making raw into general interest so maybe four or five hours of that will appear for general publication. The blackmail snippets are easy to produce; if any exists you've already received it and the adjoining demands (Foord sensibly priced his at slightly less than a trans-Atlantic plane ticket; Holden has yet to publish a price list).

* "middle school dance" is comp-sci jargon for a deadlock where party A is waiting for party B to do something and the reverse. The allusion is to boys standing on one side of the gym waiting for the girls to ask them to dance and the girls standing on the other side, etc.

High Trust, High Responsibility Communities

2010-02-26T20:48:00.005-05:00

[This is a blog version of a lightning talk I didn't give at PyCon]

People Hacking is as old as time, and every long lasting organization engages in it. One example of hacking people (and the title of this post) is to tell people you trust them. This "trust cue" invites reciprocity -- people behave well because you've signaled that you expect them to. Google's "don't be evil" motto works this way; the company has promised the outside world that it should be trusted so internal employees feel pressure to live up to the promise.

You can do the opposite and tell people that you don't trust them ["negative trust cues"] and people being people their behavior is predictably cautious and non-cooperative. I once worked for an ex-DEA prosecutor and to put it mildly the employer-employee relations weren't stellar.

The Python community has many positive trust cues going for it. The coding end is open source so everyone is a volunteer by definition, and volunteering for anything is a positive trust cue. Likewise PyCon introduces people face-to-face and having met someone as a real-live-person (even once!) invites an obligation to be more forgiving in the sterile world of email and bug trackers (was he being a dick or just having a bad day? I've met him, must have been a bad day). I got my commit bit after never having one of my many patches to CPython accepted*, but curiously not long after I met the guy who had rejected them (Raymond Hettinger, who was working the registration desk when I checked in).

I am/was a member of a large number of long standing volunteer orgs (most are fraternal**) and have seen many variations on trust cues: the big ones are the Boy Scouts, Habitat for Humanity, the Kappa Alpha Society, and the Free Masons. None are as old or put as high an emphasis on trust cues as the Masons - everything is setup to do charity and avoid conflict. It is prohibited to drink before/during meetings***, discuss politics, or discuss religion. The taboos are so strong that there aren't legal punishments for violating them - you are trusted not to violate them so (by reciprocity) no one does. Very few of the members are half as smart as the average lunch table at PyCon, but it is impossible not to like the members of my lodge because every time I see them they are doing charity: giving blood, contributing to food banks, etc. The trust cues are through the roof (one side effect of that is a very high attrition rate -- people quit because the activities are quite stolid and boring).

So the Python community in general and PyCon in particular inculcate positive trust cues (when was the last time your saw a post titled "PyCon sucked?" never). That said, I ran away from the python-d7y list because it was so loaded with negative trust cues. See Anna Ravenscroft's PyCon2010 talk for very healthy ways to promote positive trust cues and eschew negative ones. The topic mandated the opposite of the "no politics, no religion" of my other, more boring, org and the results were predictable. The d7y list was mostly harmless but 5% of the talk was by people that had no skin in the game (and hence no expectation of kindness or reciprocity). To be vague and ablative the demands of the 5% was a laundry list of negative trust cues: censorship, name-and-shame, and other minor atrocities. As a result most discussion of the list actually took place off the list because plain discussion on the list was impossible (par exemplar was one of the moderators emailing me off-book "I wish I could +1 but I can't. We need a new list.").

In closing the Python community exploits human nature in the form of trust == responsibility pretty well. I am happy to be exploited thusly because I'm human and it tickles my humanity. Assign bugs to me and - if I've met you or have met someone who has met you - I'll be quick to close it with a more delicate comment than otherwise.

* My patches weren't bad or wrong, they just weren't enough of a speed boon considering the maintenance overhead to be added to the stdlib.
** I have three brothers and no sisters so that's where I'm at ease. Though until recently (last century) "fraternal" meant "brotherhood of man" instead of the narrower "male only" so there still exist some mixed sex and female-only orgs that have "fraternity" in the name.
*** The "no booze" is very unusual for voluntary associations in the US. Most clubs have a members only bar that is the major fundraiser for the org. For instance, when I lived in Pennsylvania I was a social member of the King of Prussia Volunteer Fire Company - I was one of the thousand members who didn't fight fires but paid $30/year for my membership card (you needed to be nominated by an actual fire-fighter to join but like most orgs this was a technicality -- they were mainly interested in keeping out people who couldn't find one related person to say something nice about them)

PyCon Wrapup

2010-02-24T12:21:00.003-05:00

Here are my assorted thoughts on PyCon this year. Long story short: Marvelous.

The Atlanta location was actually in downtown Atlanta and not like the Chicago or Dallas PyCons where it was hosted nearer the airport than the city itself. There were dozens of restaurants and bars at every price-point within walking distance, and even more than that if you took the train a few stops. My favorites were the cheap local places like the Vietnamese soup joint (via Titus Brown) and the chili cheese dog joint (via Larry Hastings). I liked the high price steak joints too, but you can get good food anywhere so it's less of a kick. Now that we have some institutional knowledge about eating in Atlanta I expect to eat even better next year.

The conference organizers, program committee, and network guys all did a perfect job this year. You won't hear about it much because when things "just work" there is nothing to talk about. As an anecdote about how well things were setup: during sprints one developer was loading up his computer with video because the 'net download speed was better than he could get at home over fiber. Another anecdote: during sprints a different conference was coming in and had signs that listed our sprint rooms as part of their conference venue; I know nothing about the particulars but there was a flurry of panicked activity and the situation was handled.

Labor must be cheap in Atlanta because every business had more employees per square foot than I've seen in any other American city - including many Southern ones.

Flying sucks. While PyCon-qua-PyCon wouldn't be possible without cheap flights (trans-Atlantic esp) I would really have liked a heads-up that I would be stuck in Philadelphia for five hours. I could have called up some kith & kin and hit the downtown instead of sitting in the terminal for a dog's age.

I mostly got to talk to the people I wanted to see but haven't seen since last year (including the other Boston locals). I also met some new and interesting folks. Unfortunately one week isn't enough to do enough of both no matter how little sleep you get; but I'm pretty happy with the results.

A big shout-out to CCP games. They have an Atlanta office so an even bigger chunk of the Iceland office was at PyCon than usual. I met their Atlanta marketing director who is the guy who behind the viral marketing cum bombscare for ATHF a few years back. Good to see he landed on his feet. The CCP guys are a big reason why I slept very little, and happily so. Allow me a semi-related shout-out to Mr & Mrs Wayne & Wendy Witzel; I met them at a 'con or two ago, and Wayne volunteered as my session chair this year. They are both big EVE players and know Reykjavik better than I do (I don't know much). [Also, her name is not "Wendy" which I have been told repeatedly but refuse to acknowledge. It should be Wendy, dammit.]

Every secret project tried to do a release in time for PyCon, or so it seems. Some of the company ANNs include Sauce Labs and Nasuni. Additionally the conference was a big bag of leaky secrets so expect more announcements quite soon (I'm not contractually obligated to not-say but I will so as to not blunt their PR).

Sprints were smaller this year than last, especially python-dev. The sprint rooms at late night were also less, umm, vigorous than in past years. Sprints are still ongoing but for the first few days, when I was there, I can say there were zero card games played. Not holdem, not asshole, not set, not nothing.

I normally don't ask about late night sprint activities out of a sense of propreity, but I'll make an exception in this case. Was anyone around circa 3am on Wednesday? I have a lump on my head and I'm somewhat curious about its origins.

PyCon Prep

2010-02-16T17:02:00.002-05:00

If you are giving a talk take the time to watch AMK's How to Give a Python Talk. I posted some thoughts on speaking last year too. For anyone attending: go listen to the Pre-PyCon podcast where several regular PyConers give a conference HOWTO. It features some solid advice such as "accept the first dinner invitation you get" to which I'd add my favorite "eat lunch at a table with no one you know." It is easy to get insular especially for repeat speakers* and/or python-devs so the lunch rule helps me break out of that.

Finalize my talk and cut to length. "Finalize" is a bit of a joke because I tend to rewrite it every time I do a practice run. My talk last year got rewritten in the wee hours the night before I gave it too. Cutting to length isn't much of a problem; If you've done many rewrites and know how many slides you have you can adjust how much you talk about each slide on-the-fly. Relatedly, I'm impressed by the people that give just the talk they rehearsed - it's something I can't do.

Load up my Kindle. I'm not much of a gadget guy - I wear a watch and carry a pocket knife (I'm down to my last one, @#*@$^ TSA). The Kindle is a gift that I haven't had a chance to use because I don't travel much.

Load up my laptop. I rarely turn it on because my home office has three nice LCDs so I need to 'svn up' all the python trees and pull some bzr repositories for other projects. (I guessed wrong and started using bzr before Python announced the switch to mercurial).

Get my home network in order. I need to open up an ssh port so I can tunnel over a secure connection at the con.

Work on my lightning talks. I'm not 100% sure these will happen but be on the lookout for "High Trust, High Responsibility Communities" and "The Physics of Bowling." The community one will not include farkers, goons, or /b/tards but in my research I did come across this interesting article about EVE Online (EVE uses python heavily and always sends a contingent to PyCon) which includes a player .sig "You may be playing EVE Online, but be warned: we are playing Something Awful." My talk is about hacking people via trust cues and what does and doesn't work in some all-volunteer groups I'm in (Python-dev and Masons feature heavily).

No beta-blockers, again. I keep meaning to use performance enhancing drugs when I talk (beta-blockers are used by concert violinists to quell the symptoms of stress during performances) but never bother to procure them. Instead I just forgo coffee the day of my talk, which kinda sucks.

No mustache, again. I enjoyed it but it was a one time thing. Anecdotally: people who know you without facial hair have a hard time recognizing you with it. The opposite is not true: people who know you with facial hair have no problem recognizing you shaved.

* Jack's Rule of Badges: If two people in a group have speaker's badges then everyone in the group will have speaker's badges.

PyCon on the Charles, Night 2

2010-02-04T00:15:00.003-05:00

50+ members of the Boston Python Local turned out for PyCon on the Charles, Night 2; a dress rehearsal for PyCon speakers from the Boston area. Ned did an informal survey of the audience and 1/3rd will be at PyCon proper. I did an even more informal eyeball survey of the audience and 20% had beards and 10% were women (the groups were mutually exclusive). Only 2% of the attendees brought cookies and I would have berated her for reinforcing gender stereotypes but I was too busy thanking her for the oatmeal cookies with chocolate chips.

Peter Portante led off with his talk "The words 'Non-Blocking' and 'Asynchronous' Are Not Synonyms and Here's Why" (I paraphrase) which started with examples of documentation that use the terms synonymously and then detailed how they the two concepts are, in fact, different. The talk is a broad tour of what one thing means versus the other as well as who-owns-what as far as kernel-vs-userpace goes. Peter also footnoted the hell out of it so if you want to learn the difference between select(), epoll(), and kqueue() you can.

Glyph gave a beta* of his talk "Turtles All The Way Down" (not a paraphrase). The title references the fact that python behaves in unsurprising ways by design. His explicit goal wasn't to explain how all the "Python's Dusty Corners" work but to hammer home that what even a python newbie knows mostly holds up in strange, dark places because python's authors and maintainers are learned refugees from other camps that publish tomes like the "C++ FAQs (3rd ed)."

Gylph's talk was of particular interest to me because he's talking about the same damn topic as I am and in the same time slot. Thankfully the world is a big place so his talk is also completely different. This is my 7th PyCon so I've seen many topics repeated and I'm always pleasantly surprised by how differently the same material is presented by different people; part of it is personality, part of it is style, and part of it is voice. I, for one, would pay to listen to Alex Martelli read a phonebook.

Antonio Rodriguez gave an alpha of his keynote address. Keynotes are fun because they get to give BIG BOLD AVDICE. Hyperbole is fun because like advice from your father things need to be said twice as loud because you'll only follow half of it. The alpha talk was 3x as loud as necessary but I think he'll trim it down to the keynote normal of 2x based on feedback (and some topics will get moved to the Language Summit instead).

Edward Abrams debuted his "DJ-ing in Python: Audio processing fundamentals" talk. It is an industy talk/python success story. Not my favorite milieu but industry talks do a good job of highlighting tools and best practices. Abrams gave a brief overview of why audio manipulation is hard followed by a list of which python tools he used to solve his problems. NumPy and threads (for I/O) were featured heavily.

* version 6beta2 if his intro slide is to be believed

PyCon on the Charles

2010-01-21T00:13:00.002-05:00

The first three Boston PyCon speakers spoke last night at PyCon on the Charles, Night I. NB: If you are near Boston do show up on Feb 3rd for PyCon on the Charles, Night II with Glyph Lefkowitz, Peter Portante, and Edward Abrams.

Francesco Pierfederici did a "python success story" industry talk on using a mix of Python and Fortran-77 (via the "ctypes" module) to eat terabytes of telescope image data in real-ish time - the requirements include picking out all new asteroids from the thousands of known moving objects during the 60 second window before the next 3giga-pixel image arrives. Approximating thousands of N-Body problems in limited time is a challenge, and that's Francesco's job.

Ned Batchelder gave his talk "Go Easier on Yourself: You don't test enough and the reason you don't test enough is because you don't write easily testable code." With good examples. No, great examples. Ned is an invited speaker this year because he is concise. His talk from last year covered the same topic I have talked on before and I kept thinking "He didn't mention X" followed by "I should have omitted that too."

My talk on "Python's Dusty Corners" earned it's more accurate original twitter title "Strange Python Shit." I scrapped my originally planned "overview of everything" and 1000-slide opus that covered comparisons, descriptors, namespace lookups, and context managers. I had to because that morning I tried to put all my well researched and thoroughly unit-tested examples into power point and it came out at .. I stopped kidding myself at 40 slides and gave up. Instead I threw up the bullet point slides on what I had and just tried to talk extemporaneously; The result was about 10 minutes of speaking and 30 minutes of Q & A. This suits me just fine - as I said about Vilnius (host of EuroPython 2008) "I love this town, it is unkempt but not ruinously so."

Most questions boiled down to asking for what purpose this "strange python shit" existed and why the stuff didn't seem more useful in the generic:

Guy: what good is [protocol X] other than that it makes it easy to implement [core python feature Y]?

The answer was usually "none" which is a compelling story by itself.

Based on feedback my final talk will include more graphics (arrow pointing at the dot in "foo.bar") and include a description of why magic can happen (and why it rarely does!) at the dot. My original code examples that implement the various features like instance lookup in pure python will end up on a blog or wiki once I get them formatted. They were way too long to fit on a slide but about right for a web tutorial.

PyCon Plans

2010-01-16T11:40:00.003-05:00

My flight is booked, I'm registered, and I have otherwise been pushed, filed, stamped, indexed, briefed, debriefed and numbered*. I'll be there for the Language Summit through two and a half days of sprints.

My "Python's Dusty Corners" talk is coming along. I'll be giving a version of it next week at the Microsoft NERD Center (the too-clever acronym for their "New England Research & Development" house) at the first night of the PyCon on the Charles PyCon dress rehearsal. The talk still lacks a strong narrative - "things you might not know about python" is too loose for my tastes. I'll be polling the audience after to pick their favorite bits which will hopefully allow me to narrow the talk to one of two narratives:

Here is a feature that you don't know about and here is what it looks like when a lib/framework does use it and throws an error.

Here are all the ways that attribute lookup can be magic.

.. but if the audience feedback is that a hodge-podge of decorators, protocols, and namespace lookups actually is the interesting story I'll stick with that.

The conference organizers owe me some free drink tickets: The schedule is up and I have a good time slot (afternoon on the middle day) but I'm up against Glyph's "Turtles All the Way Down" talk. Hell, I want to see that talk. Thankfully I can because Glyph is giving a version at PyCon on the Charles night II. Feast and famine: I lucked out last year and had a packed talk because Bruce Perens was scheduled to give a keynote on the exact same topic as my talk (thus priming the pump) but then had to pull out at the last moment due to injury. Does Glyph ski? I'll have to ask him.

Be seeing you,

-Jack

* "The Prisoner" remake was terrible. I disliked it so much I felt the need to write about it on my non-python blog.

A Glimpse of PyCon in Your Hometown

2009-12-30T12:45:00.006-05:00

As in past years local Python user groups are hosting PyCon speakers to give their talks in a dress rehearsal before PyCon. It is good for the speakers, good for the local user groups, and good for PyCon (more polished talks! I did a teardown/rewrite of my talk last year based on Boston PIG feedback). Here are three dates I know of with locations and speakers:

Toronto, Feb 16, Linux Caffe

* Leigh Honeywell: Think Globally, Hack Locally - Teaching Python in Your Community
* Greg Wilson: What We've Learned From Building Basie
* Mike Fletcher: Debating 'til Dawn: Topics to keep you up all night

Boston, Jan 20, Microsoft NERD center

* Francesco Pierfederici: Python for Large Astronomical Data Reduction and Analysis Systems
* Jack Diederich: Python's Dusty Corners
* Ned Batchelder: Tests and Testability
* [added] Antonio Rodriguez: preview of his Keynote

Boston, Feb 3, Microsoft NERD center

* Peter Portante: Demystifying Non-Blocking and Asynchronous I/O
* Glyph Lefkowitz: Turtles All The Way Down: Demystifying Deferreds, Decorators, and Declarations
* Edward Abrams: DJing in Python: Audio processing fundamentals

Seattle, Jan 30, Paul Allen Center

[added from Seattle PIG in the comments]
* Speakers TBD

The Toronto announcement is here and though Boston hasn't been officially announced it was organized by Ned Batchelder this year (he organized last year's "PyCon on the Charles" too).

I'm happy to see Boston back up in the rankings with six talks this year. I intend to fully [ab]use my slot by briefly highlighting all the "dusty corners" (all the __method__ methods and object protocols) and then, based on feedback, expanding on just a subset for PyCon proper. The opportunity to see talks beforehand is a boon; you can see a talk NOW and free up a slot during PyCon and you can also see talks that you would otherwise skip altogether (I saw a great talk last year that I wouldn't have picked based on the printed program's blurb). And of course you can get ahead on your "hallway track" by catching up with people a month ahead of time.

Cthulhu and Python

2009-10-05T20:28:00.003-04:00

Andrew Kuchling broached the topic on twitter so I think that makes it fair game.

For those of you who don't know H.P. Lovecraft wrote the short fiction horror story The Call of Cthulhu in 1926. The story would only become popular much later, and only in recent decades it has been every sci-fi writer's rite of passage to embellish and extend the genre. It is to those authors what The Aristocrats joke is to stand-up comedians - everyone tries to take the same idea and make it their own. Like The Aristocrats the Cthulhu Mythos is widely pursued because it is opened ended (inviting mutations) and nobody gets sued for riffing on it.

There are some sci-fi authors that routinely include Python in their fiction. Charlie Stross is one of those; as it happens at a PyCon I was chatting with a developer who was extolling Stross's work (his girlfriend happened to be DoD and working on a counter virus predicted by Stross and which Stross named in his book after a 1967 experimental Swedish soft-core movie). I had recently read and enjoyed Stross's short variation on the Cthulhu mythos A Colder War (in which Ollie North and Reagan damn us all to Hell using not nukes, but instead weakly godlike beings) and I've been a Stross fan ever since.

Charlie has featured python-3000 in his last couple books, the first of which was put out when py3.0 was just a joking reference. His politics are plain enough (though Scottish he's more of a Fabian Socialist) but his writing is tight so the occasional implausibilities (python dominates the world; the US economy becomes third world in ten years) are forgivable.

I would have tiwttered all that, but the medium doesn't allow it.

My PyCon 2010 Talk

2009-09-30T21:10:00.003-04:00

My talk is tentatively titled "Python's Dusty Corners."* It will be a brief overview of all the features in python that you don't need to know about right up until the moment you do. The list includes how comparisons work, descriptors, iterators, context managers, namespaces, else clauses on for/while loops (suggested by Hettinger), and whatever else you can suggest in the comments (please do!). The narrative of the talk is that these are features that you don't need and/or shouldn't use in your day-to-day code but that you need to keep in the back of your mind because other people's code and the stdlib do use them. As Alex Martelli pointed out in his wonderful talk Abstractions as Leverage you can't successfully function at one level of abstraction if you don't know what is going on at the next level down. This talk is a whirlwind tour of the next level down.

I'm honored to be an invited speaker this year. This just means the program committee has pre-approved any talk I give instead of going through the normal program committee proposal process. I had some stem-winding conversations with friends about what this means and what purpose it serves. Firstly it is a flattering inducement to get prior popular speakers to speak again, and possibly no more than that. Secondly it gives speakers a chance to do a talk that might not make it through the normal approval process. I knew a person in college** who's motto was "There is a fine line between being The Man and being That Guy." Imagine a Venn diagram with barely overlapping circles labeled "Good Ideas" and "Bad Ideas;" Being "The Man" is the thin overlap between the two, and committees are very good at avoiding any idea that is anywhere close to the "Bad ideas" region, let alone one that that is actually in it. Having invited speakers is a way for the committee to include those ideas with minimal risk by inviting people who have a proven track record and hoping they don't screw up.

That said my talk is pretty safe and certainly would have made it through the normal process. I would love to give a talk I thought was in the dangerous "The Man" zone but I haven't the foggiest idea of what that talk would be. Err, I have some idea but none long enough to be a proper talk. For lightning talks I'll be preparing "I love graphs" (I do, and I have the graphs to prove it), "The Physics of Bowling Balls" (waaay more interesting than you would guess), and my always-threatened-never-done talk "PyAsshole: Simulating a partial information, non-trump, drinking card game in Python."

* As much as I liked the title I proposed on twitter It wouldn't help the conference (or me, or anyone really) to have a talk titled "Strange Python Shit" on the program.
** He looked suspiciously similar to me, but with hair down to his shoulders and an eyebrow ring (lay off, it was the early 90s).

Your Talk Proposal Here

2009-09-30T00:55:00.004-04:00

As it turns out one of the marginal items that gets trimmed in a down economy is conference jaunts [Q: who would have guessed? A: everybody]. PyCon 2010 needs talks, so if you have something interesting to say to a few hundred people this is your chance. I didn't volunteer for the program committee this year so I don't know the exact numbers but the acceptance rate is going to be much higher than the 50% for past years.

This is a great opportunity to practice your chops; I got my start when PyCon was 300 people and talks had a 90% acceptance rate. The last few years PyCon has had 1000+ people and even the smallest talk room gets 200 people. If you have anything interesting to say, this is your chance to say it.

NB, hopefully PyCon won't have to do what other conferences routinely do and say "we've extended the deadline, but this time we mean it!" For fuck's sake the conference is in Atlanta in February - I'll be happily golfing during the time I'm not speaking.

NB, by "golfing" I also mean "going to the shooting range" and "bowling" as weather dictates.

ENB: Anna Ravenscroft has a tidy short list of bogus reasons why you can't give a talk. The title is pointed at women but the excuses are universal.

Looking For a New Gig

2009-08-21T14:54:00.003-04:00

I'm a Boston based python developer looking for a new gig, full time or consulting. Here are my resume highlights:

Co-founded a web analytics & marketing company (Psynchronous Communications), and grew it to 1M+ in annual revenues.

Python core developer, author of Class Decorators.

PyCon, EuroPython, and PyCon UK speaker. (video of my PyCon2009 talk).

Hired, managed, and led small teams of developers in the rapid development of web applications.

10+ years at web startups using a LAMP stack. 7+ years in Python. 10+ years C/C++.

OpenAir.com company foosball champion.

The full "brag sheet" available on request (which omits that last item due to space restrictions).

Jack Diederich
email: jackdied@gmail.com
cell: 617-821-1734

PS I've posted my contact information in several public places over the years but it is still invisible to google. Hopefully repetition will fix that.

Dominoes

2009-07-12T20:32:00.009-04:00

Playing dominoes is hard. Like playing card games there are many games you can play with dominoes and last night I was exposed to Muggins/Fives. Muggins has a points system and unlike the kiddie version of dominoes that I grew up with is not a game of chance (think War versus Bridge). The big tipoffs that it is a hard problem are that A) it is a partial information game [you can only see the board and your own hand] and B) grown men play against each other for money [I'm told the variant we were playing is very popular in the Bahamas].

The first thing I did was look for AI research papers on solving dominoes. After an hour of feeding terms to search engines I can say: there is no research. This is strange because Checkers (a boring perfect information game) was still interesting enough to researchers that it was solved only 10 years ago. People don't bet on checkers so the fact that dominoes is a research orphan left me intrigued. Of course there's a personal angle too: If it was an easy problem to solve I could solve it and then travel while grifting strangers out of their money; If it was a hard problem to solve (like Bridge or Go) then I'd add it to my list of "fun things to play."

In Muggins you get points when all of the exposed end tiles add up to a multiple of five. If you go out first you also get bonus points equal to the sum of the pips in your opponents' hands.

After a couple hours I had a working 150 line simulator and two simple strategies: play any legal move (dumb strategy) and play the highest scoring legal move (less dumb). I use the term "working" loosely. A number of nefarious bugs related to non-randomness lurked in the code. These caused the last player added to win ties in scoring, who goes first, and some other spots. You can view the final dominoes.py source here.

To implement the strategies I went for the simplest thing that could possibly work: generator coroutines. They have a simple interface and they keep state so even somewhat complicated strategies are possible without writing a big interface class. You just write a function with a couple breakpoints and everything just works. Here is the generator for "play the first move that is legal"

def dumb_player(player_no, board, hand):
    ''' play the first domino that is legal '''
    yield None # signal that we are setup
    while True:
        for a, b in pairs_and_reverse(hand): # (a,b)+(b, a)
            try:
                board.play(player_no, a, b)
                # if we get here it is a legal play
                yield None
                break
            except IllegalPlay: pass
        else:
            # draw a possibly legal play
            board.draw(player_no)
    return

The boiler plate of setup and try/except Illegal play was identical over all the simple strategies so I refactored so that the strategy is just a scoring function applied to all legal moves. The simple strategy becomes ("a" and "b" are the two ends of the domino, "a" is the side that matches the board and "b" is the open end left after play):


def dumb_player(board, player_no, a, b):
    ''' randomly choose a legal play '''
    return None # all plays are equal

.. and the strategy for playing the highest scoring move possible is


def score_player(board, player_no, a, b):
    ''' always plays the immediately highest scoring tile '''
    sc = board.play(player_no, a, b)
    board.undo()
    return sc

Add a function that does a round-robin tourney of all scoring functions 5000 times each and you have a quick fitness test.

Well, it turns out dominoes is easy (or at least simple). This blew away all my assumptions; If you look at the source I have a dozen scoring functions some of which consider the pips on the already played tiles and the secret pips in the players hand. The best of them beats the very simple "score_player" just 50.5% of the time. 50.5% is a solid money maker if you are playing blackjack against the house at hundreds of hands an hour, but peanuts if you are playing 10 games of dominoes an hour against someone who is using the simple score_player strategy.

The source is out there and understandable (500 lines, 300 of which are short strategy functions). If anyone can consistently beat my best attempt "score_blocker6" then post the source in the comments and I'll buy you a beer next PyCon.

ICFP Contest 2009

2009-07-05T19:51:00.007-04:00

The ICFP 2009 programming contest started a couple weeks earlier than last year, and unfortunately I only found out about it as it was ending. The challenge is to solve as best as possible in 72 hours a problem that can't be brute forced in 72 hours. I love this challenge; I've been participating in the contest (in python) since 2002 and even left EuroPython early last year just to compete. I don't do writeups every year but I did for 2007, 2004, and 2002. This year I didn't participate officially but I did tackle the program after the fact.

The best resource for writeups and implementations is the FUN team page. They did their entry in python as did an amazingly large number of contestants this year. Their page has links to a dozen writeups, a sub-reddit, and some good pages on the maths of orbital mechanics.

The contest is a great chance to try out new tools and methodologies. I tried Unit Testing for the first time during an ICFP weekend (many moons ago) and I've been sold ever since. If Unit Testing helps you finish a time-limited competition faster then how could it possibly hurt during normal dev? I've also used the contest to try out pair programming, Test Driven Development, and pyrex [I won't talk about those more unless someone asks].

This year I used the contest as a chance to try out the ctypes module and Pygame gui library. More on that below, but first to the ICFP problem:

Problem Defined

This year, like many past years, the challenge was to implement a virtual machine that runs the binaries provided by the organizers. The VM binaries simulated a bunch of orbital mechanics. You then had to write programs that interacted with the simulations to push a satellite from one orbit to another, meet up with other satellites in orbit, and more complicated variations on the theme.
My strategy was the same as in past years:

Read the problem description, write a reference implementation in pure python, and test the hell out of the reference implementation.
[actual time: 1 hour]

Use that to write a visualizer and explore the problem mechanics and horribly underspecified written problem description.
[actual time
[actual time: 2 hours. I had to install pygame and read the docs first]

Write a work-alike in screaming fast C so as much time as possible can be spent solving the problem instead of waiting on simulations to finish.
[actual time: 3 hours. I had to learn the ctypes module and fool around with it.

The VM was very simple; it consisted of add/subtract/multiply/copy/if-test-else operations. Knocking out a working and tested version took about an hour. The tests were important because even my 150-line python implementation had a couple bugs in it. They also exposed some really shitty bugs in the written problem description. For instance a table listed 10 bits in one opcode as the 'imm' value but it should have been 3 bits of value with 7 bits of padding. The unit tests picked up bad values and a re-examination of the problem description led me to footnote 1.5 which said the value is 3-bits (why didn't they update the table too? no idea).
The nature of the VM lent itself to a fast C implementation. The executable cannot change itself so if the 30th opcode adds memory locations 101 and 102 it always adds memory locations 101 and 102. It never does a conditional jump or anything else funky. Writing a C version of the inner loop was almost as simple as adding print statements to the python version and then throwing the output at gcc. To understand the VM a little better (and test it even more) I added my own opcodes that did asserts and wrote a self-test binary that was a translation of my python unit tests into VM code. I could be confident in the C translation because it ran the self-check the same as the python version. The C version runs 1000 times faster than the pure python version, which is nice. Oddly, adding -Ox compiler flags makes the self test completely shit the bed; I say odd because the program is extremely deterministic so I guessed it would optimize nicely.

Writing a visualizer early (in pygame) was invaluable. The first draft just drew the current orbit, the target orbit, and the satellite's current position (state was printed to stdout). I assigned the arrow keys to manipulate the thrusters and discovered another bug in the spec -- setting thrust dx/dy points the thrusters in that direction so if you want to increase your speed in direction X you need to fire in direction negative X. Playing with the visualizer also answered some other ambiguities in the spec -- the simulation keeps track of your relative position to the Earth but it really means your relative position to the CENTER of the Earth adn not the surface. That isn't just important to know it also makes all the maths much easier.

Solving the level 1 problems was easy. Included in the problem description was maths for calculating thrust vectors for moving an object from one orbit to another. The maths, however, were for doing it using a minimum of fuel but the score for your solution is maximized by doing it as quickly as possible and using all your available fuel. Thanks to the first and simplier solutions I understood enough orbital mechanics to know you always wanted to fire thrusters either perpendicular or parallel to the tangent of the orbit at your current position. Thanks to the screaming fast C implementation I could brute force how much and which way to fire to get a high scoring solution. Having a fast state push/pop was a huge advantage too. Because the simulator works in discrete 1-second intervals the "real" mathematic solution is off by a little; so instead you want to do things a little early or a little later than the "ideal." That discrete solution is easily brute forced once you have the "real" solution.

Writing the ctypes/so library was interesting. The VM is specified as having a program execution area (a list of opcode, arg1, arg2 tuples), a memeory area (a finite array of doubles), a boolean status flag, a double score value, and a couple short double arrays for IO. Because the program loop is deterministic it goes away when you translate it to C. Then you are left with a bunch of double arrays of known size, one boolean, and one other double. The organizers left a big hint that you could implement this as a single array of doubles by leaving the first two values in one of the arrays undefined. So the obvious solution is to stick the one special double and the boolean flag in there (as a double) and then just concat all the arrays-of-doubles together. The max combined size is finite and under 3000 * sizeof(double). Pre-allocating a big array of these and memcpy'ing them for push/pop of state becomes dirt cheap. As a bonus it makes the ctypes interface stupid simple too because the struct is just a single array of 3000 doubles. Kinda: python doesn't have a native double type so ctypes converts to float. In order to get and set the raw double values of the VM I made the ctypes definition a union of 3000 doubles and 3000 unsigned long longs; when deciding what to do floats were close enough but when initializing the VM data or writing the trace I could set/get the 8-byte ulonglongs (hurray for c's type ignorance!).

ctypes

In the past if I wanted C speed I always hand rolled CPython extension modules. It is a little bit of extra work but the speed is unbeatable (2x faster than pyrex in my experience). ctypes is so useful I don't ever think I'll ever hand-write an extension module again. The only stdlib module that currently uses ctypes is uuid, but I expect many new modules to use ctypes instead of doing it the hard way.

The VM operates on doubles but python only has a native float type. I was very tempted to create a new core datatype by copying Objects/floatobject.c and search/replace'ing every 'float' to 'double' but because I was using ctypes I opted for the quicker and simpler casting of those doubles a ulonglongs when I needed to set/get. I bet Numpy has a way to deal with all this but I'd already hit my limit on new-tools-learned-per-hour. I understand raw C

ctypes makes easy things easy and hard things possible. If your .so (*NIX .dll) has functions that take an int/void and return an int/void you don't even need to provide a prototype -- it just works. So instead of writing a full featured Python/C wrapper for my basic datatype (an array of 64bit values) I just wrote 6 lines of python that defined the struct layout and then a C library that had a bunch of manipulation functions that returned 1/0 success/failure values and 100 lines of python that mapped property names to assignments/reads of the memory chunk. It was much less work for 90% of the speed. It also meant it was easy to apply the unit tests for my pure-python solution to the hybrid solution because the only difference was the underlying storage - a dict for the pure python and an C-array for the ctypes wrapped .so.

pygame

In the past my goto-GUI has been Tkinter. I've been using Tk every since reading Learning Python in 2001 which is half a python book and half a Tk book. I've accumulated a personal library of Tk elements that do everything from menus to graph plotting to shape drawing. I threw it all out for this year for pygame and ended up with a very decent visualizer that was just a couple hundred lines of python code. I won't be going back to Tk for graphical GUIs in the future (I still like it for text).

[More later]

ps, I use "maths" plural like the British commonwealths simply because I like it; I lived in Australia for a year and it grew on me. However, you won't hear me singularizing "sports" or saying someone is "in hospital" - they are "in the hospital." English is the best language ever because what is legal is whatever works. To paraphrase a variously attributed quote: "English doesn't borrow from other languages. It takes them down dark alleys, bashes them on the head, and rifles through their pockets." Some people will tell you English has a giant number of rules, but really that someone is just trying to make sense of a system where anything goes.

HOWTO: get useful information out of the buildbot

2009-04-07T20:04:00.005-04:00

The CPython core has a raft of machines that do nothing but pull updates from subversion (the code repository) and run the unit tests. You can see the full and somewhat cryptic list of all the boxes and their status on the buildbot webpage. I had to relearn how to read all the output because I had failing tests that only failed on other people's boxen. So here's the HOWTO.

Find your branch
Ten minutes after your checkin reload the buildbot page and find the machines running the branch you checked into. The machines are titled with the codebase being run, currently either "trunk" (aka 2.7), 2.6 (the maintenance branch), 3.0 (another maintenance branch), or 3.x (the py3k trunk). The other words in the name are some combination of hardware, operating system, and compiler.

Open a bunch of tabs
Each vertical column below the name is a time series of builds and statuses with the most recent at the top. The items are either Green (completed, OK), Red (completed, catastrophe), or Yellow (either still running, ambiguous success, or informational). Open a tab by clicking on the "Build NNN" links on all the machines running the branch you care about. Your checkin is listed in the leftmost column so only pick builds than start above (afterwards) that checkin. Then wait an hour or two. [what does the build number mean? I have no idea but I'm guessing the Nth build for that machine]

Check the builds
Most of the builds should have finished so go ahead and reload all the tabs for the individual machines. If the build is still in progress you can tell by the giant header that says "Build In Progress." If it is done you will see a series of little headers and links. Each header is for the different stages: update from svn, run ./configure, recompile the source, and run the test suite. The link titled "stdio" after each of these should be renamed more plainly "view ./configure output," "view test output" etc. This is what you want to see.

Find the output you care about
Search to find the tests and failures that apply to you. Especially on the trunk there may be failures that aren't your fault. Someone elses' checkin might even be causing an abort before your stuff even gets run. If you stuff works, great! If not..

Checkin, rinse, and repeat
Based on the output you may need to make another checkin and let all the buldbots run again. If the failure isn't verbose enough then you will have to checkin some debugging output and wait for them to run again.

.. and that's all there is to it.

telnetlib progress

2009-04-07T17:33:00.004-04:00

The first item in my Fixing Telnetlib TODO ("#1 test the hell out of telnet") is nearly done. The unit tests now test IAC handling and SB negotiation in addition to the read_* methods. As a bonus it looks like I fixed all the race conditions in the read tests'cause the builbots are going greener. (aside: did you know about Queue.join()? I didn't, very handy).

The only remaining nit is that the SB data tests are creating an uncollectable GC cycle. The Telnet object has a reference to the negotiation callback. The negotiation callback needs to call telnetob.read_sb_data() to get at the SB data. So I have a nego_collector class that looks like


class nego_collector(object):
    def __init__(self, sb_getter=None):
        self.seen = ''
        self.sb_getter = sb_getter  # cycle, this is a Telnet.read_sb_data bound method
        self.sb_seen = ''

    def do_nego(self, sock, cmd, opt):
        self.seen += cmd + opt
        if cmd == tl.SE and self.sb_getter:
            sb_data = self.sb_getter()
            self.sb_seen += sb_data

The nego_collector either needs to keep a weakref to the function or we have to break the cycle manually. Consider this just another crufty corner in telnetlib.

[woops]. I spoke too soon. Not all the buildbots are passing so I now have a machine running the telnetlib tests in an infinite loop with the CPU heavily loaded. Hopefully I can smoke out the remaining race conditions locally. If not I'll have to sign up to use the Snakebite testing farm.
[later] Fixed. Almost certainly. We now allow a margin of error of 100% (a whopping 0.3 seconds) in our timing assertions and we do fewer of them.

Speaking about Speaking

2009-04-04T22:41:00.004-04:00

AMK's talk How to Give a Python Talk is very informative, you should watch it even if you aren't planning on giving a talk. Why should you watch it? partly because it gives you an idea of what goes into a talk and partly because it demystifies giving a talk enough that it might prompt you into giving one. Lots of solid advice.

Andrew's talk itself is a nice illustration of some of his points. No one would mistake Andrew for a motivation speaker; you don't walk away from that talk with an inexplicable need to buy what he's selling and given the audience you might actually be pissed off if you thought he was trying to sell something. (talk->content != NULL) ? Good_talk : Bad_talk. PyCon attendees care more about red meat than glitter and are very forgiving on presentation if the red meat is there.

How I do it what I do to prepare has heavy overlap with what Andrew recommends. Practice is king. When I step on the stage I'm not nervous per se, but when speaking in front of a large audience I do tend to read the slides much more than I talk about them in practice. So my rule of thumb is to practice a talk where I spend three minutes per slide knowing that I'll drop most of my segues and only spend one minute live talking per slide. Figure out your own constant and practice against that. I was amazed at Ned Batchelder's talk because the the video of his talk matched so closely with his text explication of his slides. The prepared text is almost 1-to-1 which I personally just can't do.

Narrative, Narrative, Narrative: Pick a theme and stick with it. If you don't talk to your premise once every couple minutes then you have failed. My talk was "Class Decorators: Radically Simple" and I tried to say on every example that a decorator was a callable that took one argument and returned something. Raymond Hettinger's talk was "Easy AI in Python" and he started and finished every example emphasizing that a novice could do it. Alex Martelli's talk was "Abstractions as Leverage" and he introduced every slide with a quote from a very dead (and sometimes white) male who had made the same point back when writing was a novelty. It seems odd but part of your job as a speaker is to repeat yourself, repeatedly.

Don't drink coffee: This sucks, but you can't drink your normal amount of coffee before your talk. I was hoping to drink a few cups and balance it out with a bloody mary but my talk was in the AM and the hotel bar wasn't open. Instead I drank only a little coffee so I wouldn't be humming on stage. I'm told Beta Blockers work to suppress the nerves (symphony orchestras use them) but I haven't tried it myself.

Practice is free and Plentiful: It is a not-so-secret fact that user groups, PIGs, and even Cons are starved for presenters. My most recent talk started as a lightning talk and then I gave it at a local user's group and a couple Cons that had 90%+ acceptance rates before giving it at PyCon. Practice is good and the opportunities for practice are many.

You already know something to talk about At the Boston PIG talk-dry-run (all the PyCon presenters gave their talk to 30 people a week before they gave it to 300+) I spent the first five minutes talking about talking. You do know something you can do a talk about and it sounds like "what is something I wish I knew about one year ago?" It's that easy. Try one or three ideas on the local group as a lightning talk and then grow the best one into a proper talk proposal.

It isn't complicated, see you with a speaker's badge next year!

Small test_telnetlib progress

2009-04-04T21:53:00.002-04:00

My first patch of test_telnetlib is up. It tests most of the guarantees that the various Telnet.read_* methods make (I'm sure I missed a couple). The only problem is that every single test theoretically has a race condition. In actual practice the chances of a race are 0.0%, but theoretically it isn't sound. I posted it as a patch (as opposed to just committing it) to see if anyone has an opinion.

For the next round of tests I'll be writing unit tests for the out-of-band negotiations parser.

PyCon Errata

2009-04-03T16:59:00.007-04:00

Old and New Faces It was good to see everyone, too many names to mention. That includes all the other Boston pythoneers who I tend to see just once a year and in a city not named "Boston." There is never enough time to time to talk to everyone but I did try. I also did my usual thing which is to purposely eat lunch with no one I know [it's my fifth PyCon so this rule has been relaxed to "as few people I know as possible"]. A few mentions: somehow I'd never met Jesse Noller before (despite many PyCons and him being in Boston); Georg Brandl made it over to the US for PyCon for the first time; I didn't run into Martin Blais until day five when he was sitting next to me at sprints; a sixteen year old (who is senior to me on py-dev) thanked me for contributing a patch; and David Mertz (whom I had never met in person) ran up, introduced himself, and disappeared into the ether (far too brief: I have to invite him over for dinner or something).

Limited Excess In a down economy attendance and freebies were also down. Almost no speakers ended their talk with a "and we'ere hiring!" slide as opposed to the past standard of 100%. To my shock and horror I actually had to pay for most of my own dinners and drinks. CCP/EVE Online was a standout in this respect [If you're wondering how a company in Iceland can afford to be generous remember that their subscribers pay in dollars and euros, not kronas].

EVE Fan-Fest I learned about EVE Fan-Fest not from the CCP guys but from a husband/wife team of players. 1500+ gamers descend on Reykjavik annually. This is such a large number of extra people for a country of 300k that the conference has to be closely coordinated with the government, hotels, and airlines. The mind reels.

Code Blindness By the end of sprints I was suffering from the geek equivalent of snow blindness. Throughout sprints I traded bug reports, emails, and checkins with Hiro Yamamoto (the "John Smith" of Japan). He'd miss something and I'd whargarbl his name under my breath. I'd miss something and know he was grumbling half way across the world. I pretty clearly lost that battle when I committed a patch that checked to see if unsigned longs were less than zero (oh sure, the compiler can optimize it out, but still..). Which reminds me, I still need to revert that.

We have a prodigy on our critical path. Python's release manager is Benjamin Peterson and Benjamin is sixteen years old. On the internet nobody knows you're a dog and in open source no one cares if you're in High School. He gets stuff done, end of story. There is a small amount of cognitive dissonance involved, but not much. For instance he gave me an attaboy for a patch I submitted last year - and while I have shoes that are older than he is - he sincerely meant it as a compliment and I took it as such. He's good people to have around - though if he gets a driver's license or a girlfriend we're in a spot of trouble. [I talked to his mother only briefly but she treated his hobby as casually as if he was on a sports team.]

Benjamin is not without precedent. Our now somewhat older prodigy is named Georg Brandl. The idea of prolonged adolescence is pretty new in cultural terms (less than 60 years old). Both men are sterling illustrations that when you treat "kids" like adults, they behave like adults (heck, they were adults in the first place but just not acknowledged as so). Let's have more of this please.

Twitter Twitter was the breakout story of the year at PyCon. I've peeked at it several times but never seen the point. I'm so old school I still refer to IM as "talk." Twitter was nowhere to be seen last year but this year it was pervasive. Sure, most of the tweets were mindless blather but they fill the mindless blather niche very well. "bourbon in the Kennedy room" is useful when broadcast but not the kind of thing you'd send an email about. Michael Foord (aka voidspace) gained 50 followers a day during the conference. I have reluctantly broken down and signed up too. Oddly one of my first tweets was answering the question "do I need stitches for this?" which is something I know much about (I had a very full childhood and I have the scars to prove it).

My Talk Video of my talk Class Decorators: Radically Simple is now online. I was pleased with my performance until I saw the video. Thankfully attendees care more about content that presentation because there are a dozen things I would like to do over; I don't have a future as a motivational speaker. I have done a talk on that same topic several times now and this time was a giant rewrite. The night before I was in bed by midnight but tossed and turned. I ended up giving up and rewriting large portions until 5am. I slept for three hours and what you see was me looking at the slides for the second time. All the ridiculous example slides were what people [unsolicited!] came up and told me is what made class decorators "click" for them. Go figure.

There is a raft of little things I would change about the presentation. Unfortunately I won't ever give it again so I'll have to apply them to my next talk (after I think one up). Bloused shirt? gone, starch that thing and make sure it is tucked in. Conversational voice? gone, I have a separate speaker's voice and I didn't use it (lack of sleep?). USB remote slide dongle? gone, I spent as much time aiming the laser pointer at the screen as I did talking to the room. Wireless mike? keep, standing at the podium sucks [I lucked out - I was in the only room that had a wireless mike and I only got to use it because I asked].

Oh, and the perenial "pause between sentences." For the first five minutes I talked like I was reading a teleprompter. There isn't much you can do about this other than practice.

[and then some more errata]

International As I've mentioned before PyCon is the inverse of EuroPython in that it is 75% American and 25% European (eyeball numbers: I'd love to see hard data on this). The speakers list is somewhat more static because there is a subset of people who go to conventions for fun (myself included). To confuse things further there are a number of Americans who weren't born here and some "Americans" who are American but not in name (Alex Martelli is still Italian for sentimental reasons despite living in and literally marrying into to America).

Martelli's Slides Alex Martelli's slides are immediately recognizable because he uses the same background and the same quirky font on all of them, always. I got the scoop from Anna Ravenscroft (a sometimes PyCon speaker and AKA Mrs Alex Martelli). He is fond of the background and font because they remind him of a blackboard. No one has complained so that's all there is to it.

Sprints are Magic Two days of sprints generated the same amount of python-checkin traffic as a regular month. Questions are just so much cheaper in person than in email that it couldn't be otherwise. Raise you hand and say "can anyone tell me about [interface]" and you get an answer. Person-to-person social pressures also lead to quicker bug resolution. Jesse Noller said something like "I assigned a pickle functools bug to you while you were in the can, it seemed up your alley." It wasn't up my alley but a few hours later I had read the pickle docs and checked in a patch to make functools.partial instances pickle-able.

Fixing telnetlib

2009-04-03T14:54:00.004-04:00

During the PyCon sprints I re-assigned all open (and unclaimed) telnetlib bugs to myself. The biggest longstanding complaint about telnetlib is that non-trivial negotations aren't possible because the negotiation callback is very bare bones. The biggest problem with telnetlib is that there is almost no test suite - which is why some bugs have been open for seven years. So my priorities are first to test the hell out of telnetlib and second to improve negotiation.

The negotiation problem is clearest when dealing with two-way communications like NAWS (Negotiate About Window Size). The first time the server asks DO NAWS the client can reply WILL NAWS and include its current window size. The current negotiation callback supports this just fine. But when the client resizes its window it needs to be able to tell the server, which means Telnet needs a hook for a pending negotiations queue. And forget about the STATUS code which asks the other end of the connection to say what options it thinks have been negotiated - the current Telnet has no notion of state.

Below are the raw TODO and research notes I put together in a few hours at sprints. I used google code search to find some of the attempts to fix telnetlib by either subclassing it or writing a semi-compatible Telnet-alike from scratch (these are harder to grep for, for obvious reasons). The RFCs section marks each RFC as Must/Will/Won't implement. "Must implement" means core stuff for the Telnet class, "Will implement" means the telnetlib should include a negotation implementation for that RFC, and "Won't implement" means it won't (because the RFC is either archaic or otherwise unused in the wild). The BUGS list includes all open bugs and the closed bugs I want to revisit or double-check.

---- TESTING TELNETLIB ----
* Testing
- test the read_* gaurantees
- test timeouts (already implemented?)
- test the sb handling
* make real negotation possible
* add real timeout and prompt exceptions
* make Telnet objects context managers
* process_rawq is a train wreck. Make sure we do something compatible but less icky.
* figure out where the hell they found all those contstants.
* Why is chr(17)/"\021" blindly filtered out of the stream?

---- BUGS ----

OPEN

http://bugs.python.org/issue5188
telnetlib process_rawq buffer handling is confused

http://bugs.python.org/issue2550
SO_REUSEADDR doesn't have the same semantics on Windows as on Unix

http://bugs.python.org/issue1360221
telnetlib expect() and read_until() do not time out properly

http://bugs.python.org/issue1252001
Issue with telnetlib read_until not timing out

http://bugs.python.org/issue1049450
Solaris: EINTR exception in select/socket calls in telnetlib

http://bugs.python.org/issue708007
TelnetPopen3, TelnetBase, Expect split
[THIS, a rewrite of telnetlib. Mine for good stuff]

http://bugs.python.org/issue1678077
improve telnetlib.Telnet so option negotiation becomes easie

http://bugs.python.org/issue1772788
chr(128) in u'only ascii' -> TypeError with misleading msg

http://bugs.python.org/issue1737737
telnetlib.Telnet does not process DATA MARK (DM)

http://bugs.python.org/issue1772794
Telnetlib dosn't accept u'only ascii'

CLOSED

http://bugs.python.org/issue2451
No way to disable socket timeouts in httplib, etc.

http://bugs.python.org/issue822974
Telnet.read_until() timeout parameter misleading

http://bugs.python.org/issue630829
telnetlib.py: don't block on IAC and enhancement

http://bugs.python.org/issue723312
ability to pass a timeout to underlying socket

http://bugs.python.org/issue1520081
telnetlib.py change to ease option handling.

http://bugs.python.org/issue664020
telnetlib option subnegotiation fix

http://bugs.python.org/issue723364
terminal type option subnegotiation in telnetlib

---- RFCs ----

http://en.wikipedia.org/wiki/Telnet
Wikipedia lists all the relevant RFCs at the bottom.

[--FORMAT--]
URL
Short Description
Will/Won't implement
[--FORMAT--]

http://www.iana.org/assignments/telnet-options
List of officially assigned option codes
Must implement.

http://tools.ietf.org/html/rfc854
(1983) Telnet protocol definition.
Must implement.

http://tools.ietf.org/html/rfc855
(1983) Telnet negotation.
Must implement.

http://tools.ietf.org/html/rfc856
(1983) Telnet binary protocol.
Won't implement. This was obviated by Kermit, Zmodem, and the like.

http://tools.ietf.org/html/rfc857
(1983) Telnet ECHO negotiation.
Will implement.

http://tools.ietf.org/html/rfc858
(1983) Supress Go-Ahead. Nego supression of "your turn" messages for full duplex connections.
Won't implement.

http://tools.ietf.org/html/rfc859 (Obsoletes http://tools.ietf.org/html/rfc651)
(1983) Telnet status. Ask other party to retransmit what they think the current negotiated options are.
Will implement.

http://tools.ietf.org/html/rfc860
(1983) Timing mark. A work around for servers that can't read the socket as fast as people type (!!!).
Won't implement.

http://tools.ietf.org/html/rfc861
(1983) negotiating about negotiating
Proln't, Doubtful this is still in effect.

http://tools.ietf.org/html/rfc885
(1983) End-of-Record code.
Might, I have a vague recollecting that this is used as a prompt sigil.

http://tools.ietf.org/html/rfc1073
(1988) NAWS (Negotiate About Window Size)
Will implement.

http://tools.ietf.org/html/rfc1079
(1988) Baud rate negotiation
Won't implement.

http://tools.ietf.org/html/rfc1091 (Obsoletes http://tools.ietf.org/html/rfc930)
(1989) Terminal type negotiation
Will implement.

http://tools.ietf.org/html/rfc1184 (Obsoletes http://tools.ietf.org/html/rfc1116)
(1990) Telnet linemode nego. Basically save packets by being less interactive.
Won't implement.

http://tools.ietf.org/html/rfc1372 (Obsoletes http://tools.ietf.org/html/rfc1080)
(1992) Terminal flow control. Local terminal stuff.
Won't implement.

http://tools.ietf.org/html/rfc2217
(1997) SLIP-lite protocol for sharing a modem.
Won't implement.

http://tools.ietf.org/html/rfc2946
(2000) Telnet Encryption nego.
Won't implement (does anyone actually use this?)

http://tools.ietf.org/html/rfc4777
(2006) IBM iSeries hardware telnet extensions.
Won't implement (starngely, the RFC argues against implementing itself)

---- Alternate Implementations ----
[found using google code search]
a hacky ECHO negotiator

subclass-and-patch NAWS negotiator

a from-scratch wrapper

a from-scratch reimplementation w/ better (but unpythonic) negotiating.

PyCon Organizers: Doug loves numbers and numbers love Doug

2009-03-31T03:11:00.004-04:00

The PyCon organizers struggle to make each conference better than the last. As I mentioned in another post they did a bang up job on logistics this year. One metric they track is speaker popularity. It is a bit fuzzy because hot topics vary and individual speakers can do well on one topic while sucking at another. But the organizers do try to sift out the best.

Speaker Data

This year generated a bonanza of speaker data. The online talk schedule was all interact-y and allowed attendees to plan and print their preferred talks ahead of time. During the conference itself the back of every room had a pile of poker chips and three buckets: Green, Yellow, and Red. The idea being that everyone drops a chip in the Good/Neutral/Bad bucket as they walk out at the end of each talk [hopefully they leave at the end].

Doug Knowns Data

If you were thinking the stats are weak and there are many ways to game the numbers you are very right and very wrong. Doug Napoleone loves data even more than I do so he's doing regressions like nobodies business. Doug has a post up explaining the raw data and the problems associated with turning the raw stuff into usable numbers. He does speech recognition software as his day job so he knowns statistics backwards and forwards.

Is This Pythonic? Conclusion

2009-03-31T02:35:00.002-04:00

I couldn't leave well enough alone and continued to refactor the code from the Is This Pythonic? Open Space. The final version I submitted as a patch to the original project. The biggest change was not in apply_all (as I assumed) but in writing a new chunk of code that sucks all the ugly and special cases from the rest of the code and puts it in one place. I don't know if there is a pattern name for this but it tends to happen at boundaries. Pretty print functions are usually ugly for instance, and for good reason - your only other choice is to ugly up the core.

So here is FetchAccumulator. It sits at the boundry just above the database calls and returns tidy, regular data to its callers.


class FetchAccumulator(object):
  def __init__(self, sql, args=None, fetch_per=-1, limit=-1):
    self.results = []
    self.sql = sql
    self.args = args
    self.fetch_per = fetch_per
    self.limit = limit
    return

  def fetch(self, cursor):
    cursor.execute(self.sql, self.args)
    if self.fetch_per == 1:
      results = cursor.fetchone()
      assert len(results) <= 1, results
    elif self.limit > 0:
      results = cursor.fetchmany(self.limit)
      assert len(results) <= self.limit, (len(results), self.limit)
    else:
      results = cursor.fetchall()

    if not results or not filter(None, results): # code smell
      return

    self.results.extend(results)
    self.limit -= len(results)

    if not self.limit: # we fetched our limit
      raise DoneApply()
    return

  def __iter__(self):
    return iter(self.results)

This makes the other functions much, much simpler. Here are four database query functions that use FetchAccumulator. Seventy lines are now twenty.


class ShardCursor(cursor.BaseCursor):
    def selectOne(self, sql, args=None):
        accum = FetchAccumulator(sql, args, fetch_per=1, limit=1)
        apply_all(valid_shards(self._shard), accum.fetch)
        return accum

    def selectMany(self, sql, args=None, size=-1):
        accum = FetchAccumulator(sql, args, limit=size)
        apply_all(valid_shards(self._shard), accum.fetch)
        return accum

    def selectAll(self, sql, args=None):
        accum = FetchAccumulator(sql, args)
        apply_all(valid_shards(self._shard), accum.fetch)
        return accum

    def countOne(self, sql, args=None):
        accum = FetchAccumulator(sql, args, fetch_per=1)
        apply_all(valid_shards(self._shard), accum.fetch)
        return accum

Of course these functions now have their own code smell -- they only vary in their accumulator so they could be collapsed into a single function. That would require refactoring all the calling code which is a bigger project than I wanted to take on.

The apply_all function grew a proper exception to allow callers to bail out of the loop early.


class DoneApply(Exception): pass

def apply_all(shards, func):
  for shard in shards:
    db = shard.establishConnection()
    try:
      cursor = db.cursor()
      func(cursor)
    except DoneApply:
      break
    finally:
      db.close()

I'll omit the unit tests. The original project had no unit tests for this code so I had to write some to make sure my refactoring wasn't breaking anything.

Is This Pythonic?

2009-03-29T17:01:00.007-04:00

Moshe Zadka and I did an Open Space titled "Is This Pythonic?" where we took someone else's code and reworked it to be cleaner. The code we worked on was cursors.py from the PyShards project.

[Originally Steve Holden and Raymond Hettinger were going to host it (they've done it before) but Steve bowed out and Raymond decided to go downtown with his girl]

Here is the selectOne function in it's original form.


def selectOne(self, sql, args=None):
  results = []
  shard = self._shard;
  while shard != None and len(results) == 0:
    db = shard.establishConnection()
    cursor = db.cursor()
    cursor.execute(sql, args)
    res = cursor.fetchone()
    if res != None:
      results.extend(res)
      cursor.close ()
      db.close ()
      shard = shard.next
  return results

The code mixes a bunch of conceptual actions in one big blob. It is walking a linked list* of shards. It acquires a resource (making it harder to test) but doesn't safely release it in a try/finally. It builds up a list of results, and finally returns it. That's a lot of things for one function to be doing at once.

Below was the first cut. Each action is broken into a separate function. Because there are many functions almost like this one we can even reuse those parts.

* The linked list should just be a list, but that's a bigger refactoring.


def valid_shards(shard):
  ''' walk the shards linked list, yielding the items '''
  while shard:
    yield shard
    shard = shard.next

def apply_all(shards, func):
  ''' for each shard connect to the database, create a cursor, and pass it to func '''
  for shard in shards:
    db = shard.establishConnection()
    try:
      cursor = db.cursor()
      yield func(cursor)
    finally:
      db.close()

def selectOne(self, sql, args):
  ''' execute sql on each shard, returning the first row (if any) on each shard'''
  def fetchone(cursor):
    return curser.fetchone(sql, args)

  results = apply_all(valid_shards(self._shard), fetchone)
  return filter(None, results)

So each function has a little job and does it in a straghtforward way. Because the module has many methods that are almost like selectOne() we should be able to reuse those parts. So we gave it a try on selectMany()


def selectMany(self, sql, args=None, size=None):
        results = []
        stillToFetch = size
        shard = self._shard;
        while shard != None and stillToFetch > 0:
            db = shard.establishConnection()
            cursor = db.cursor()
            cursor.execute(sql, args)
            res = cursor.fetchmany(stillToFetch)
            if res != None:
                results.extend(res)
                stillToFetch = stillToFetch - len(res)
            cursor.close ()
            db.close ()
            shard = shard.next
        return results

SelectMany has an extra wrinkle that SelectOne doesn't in that it will stop early if it gets enough result rows. The apply_all function doesn't have a hook for stopping early so we have to kludge one into the function we pass in. Here is the first draft that has a big code smell. Raising StopIteration will do the right thing but it won't if the implementation changes.


def selectMany(self, sql, args=None, size=None):
  limit = [size]
  def fetchmany(cursor):
    res = cursor.fetchmany(sql, args)
    limit[0] = limit[0] - len(res)
    if size is not None and limit[0] <= 0:
      raise StopIteration
  
  for results in apply_all(valid_shards(self), fetchmany):
    for result in results:
      yield result

This code would be much cleaner in python2.6, and much much cleaner in 2.7 (the dev trunk). So let's pretend that 'nonlocal' and 'yield from' are available.


def selectMany(self, sql, args=None, size=None):
  def fetchmany(cursor):
    nonlocal size
    res = cursor.fetchmany(sql, args)
    size -= len(res)
    if size and size <= 0: # our bug just became more obvious!
      raise StopIteration
  
  for results in filter(None, apply_all(valid_shards(self), fetchmany)): # bug fixed!
    yield from results

[I fixed the missing filter bug in that one too]
Let's fix that size bug and raise a specific exception so our code is safe even if the implementation of apply_all changes.


def selectMany(self, sql, args=None, size=None):
  class LimitReached(Exception): pass
  def fetchmany(cursor):
    nonlocal size    
    if size is not None and size <= 0: # bug fixed!
      raise LimitReached
    res = cursor.fetchmany(sql, args)
    size -= len(res)

  try:
    for results in filter(None, apply_all(valid_shards(self), fetchmany)):
      yield from results
  except LimitReached:
    pass

Yuck. That might more correct but now the code smell is stinking up the room. What we need to do is stuff more smarts into apply_all().
To Be Continued...

PyCon Organizers

2009-03-29T16:16:00.002-04:00

The organizers did a bang up job this year. With the addition of a green room for speaker prep (coffee available all day w/ your speaker badge) and walkie-talkies for the organizers everything went smoothly. I heard a couple gripes about the Wifi but I haven't had any problems myself.

The hitches are with the venue and are the usual complaints. They charge $25/day for wifi in the hotel rooms. We don't have projectors in the open space rooms because they want $600/per projector per day. Coffee and beverage service gets torn down and put back up repeated so they can charge each time. Annoying.

The videos for the conference are already being posted. This is an amazing feat - each talk has three video and several audio channels that have to be spliced together. I'll link when mine goes up.

Thanks guys!

Open Spaces Board is teh Funny

2009-03-29T11:10:00.004-04:00

Someone got drunk and clever and backfilled yesterdays Open Spaces schedule board with fictional talks [typos mine, I was touch typing]. Another guy is making a panorama photo, I'll add a link when he sends it to me.

Teach Me Ian Bicking.
Settings.py: Why sysadmins love editing your .py files.
A.N.U.S.: Plugabble Sphincters
Traversal: URL mapping is for the west.
Forking: because arguing is too hard.
Djylons: Let's make it happen.
Fulton v Rossum: Cage Match ($10)
GROIN: Come see how it works.
Zope: Making the simple IMPOSSIBLE.
Plone: Making Zope unreadable.
Acquisition Algebra: Fultonian mind fuck & other OOPSLA oddities.
Tic Tac Toe: Learn how to play, learn the secret strategies.
Tresling: Arm wrestling + Tetris, let us teach you!
Wheels: let's try them square (w/ pic of a trapazoid).
Niagra Planning: Waterfall 2.0 session
Catastophe Planning: Waterfall 3.0 session
Play: Rubix cube with a brown belt.
Reality: My hairy twisted pony.
Pickle: Love/Hate.
Let's get the hell out of Rosemont and find something decent to eat.
re Rosemont: STEAK.
ISO 9000: The future of python?

PyCon Day2

2009-03-28T15:37:00.008-04:00

Class Decorators: Radically Simple

I gave my talk today, slides are available on the PyCon website (the ppt version might be crap - I exported from OpenOffice). I added two pages of speaker's notes at the top that answer some questions (and whargarbls) and some eratta.

The final talk only shares a few slides with the PyConUK version and maybe none with the original EuroPython version. I went to bed at midnight like a good boy but tossed and turned thinking of the talk until 2am. At that point I gave up and rewrote a big chunk of the talk (which had already been rewritten since Boston two weeks ago). I finished around 5am and then sacked out for 3 hours. Somehow it managed to come in at the perfect length of 25 minutes (+5 for Q/A).

Now that I have this talk licked, I'll be retiring it. I have about 8 months to come up with an idea for next year.

Talks

[updated as I go to them]
Manfred Schwendinger: "Google App Engine: How to survive in Google's Ecosystem"
This was a detailed description of how his particular application uses cloud services (they use both Amazon EC2 and Google AppEngine). It was interesting but very detail oriented (we do this like this, and that like that). I'll be downloading the slides for future reference.

Bob Ippolito: "Drop ACID and think about data"
Bob's talk was about using non-ACID (basically non-SQL) storage. A good overview of why you, the web developer, probably don't care about the things ACID databases do well and don't care about the things alternate data stores (key-value, column-based, "persistent eventually") do badly. It's a good trip through all the available alternatives. A intro pitch about each class of stores and then a quick overview of the major implementations. This talk was packed.

Ned Betchelder: "Whirlwind Excursion thrhough Writing a C Extension"
A good primer for writing modules and types in C. It's a massive subject and Ned did a good job of showing one of everything. I wasn't at the Boston meetup where he previewed his talk so I was glad to make this one.

Alex Martelli: "Abstractions as Leverage"
A typical Martelli talk, which is to say very good. His sonorous voice would tame rabid badgers. To say he's erudite doesn't begin to cover it - in 20 slides he used quotes from blogs, the American Journal of Psychriatry ("I recommend everyone subscribes"), and of course lots of dead people including a Chinese sword fighting manual (what, no Clausewitz?). Here are some of the bullets (full slides http://aleax.it/python_abst.pdf)
* to use an abstraction well you need to understand at least two layers below it
* (Splosky's Law) "All Abstractions leak", which is to say all abstractions lie.
* ex/ NFS isn't a cloal file system. It can be useful to treat it like one but if you don't know how it works you are going to get burned sooner or later.
* You can be a good python programmer without understanding how it is implemented but you can never be a great one.
* You can't write a good abstraction unless you know they layers above too -- how it will actually be used.

PyCon Day 1

2009-03-27T14:41:00.005-04:00

Today is the first day of the conference proper. The most popular talk (measured by the online talk planner widget) was canceled. Titlted "Designing Applications with Non-Relational Databases" I was sure to go, but alas the speaker canceled for reasons unknown.

This year there is a "Green Room" for speakers and conference volunteers. I wasn't expecting it to be green but I was hoping for a lounge. Instead it is a purely functional ops area - the network is run from here. It has power strips, a test projector, and free coffee. It might not have a wet bar but it is still a nice perk.

Talks

[updated as I go to them]
Brett Cannon "How Python is Developed." It was a good overview of core python development pitched at newbies. He sketched out the basic bug and feature cycles, how to [eventually] get core commit privs, etc. It was mainly an informational session so it included lost of links to the existing documentation (some of which was written by Brett).
Jess Noller "Introduction to Multiprocessing in Python." 'Multiprocessing' is a module that lets you do .. multiprocessing in python. I only new vaguely what it did before. Now I now kinda what it does. I might know more but I was busy refactoring some itertools types [see below].
Raymond Hettinger "Easy AI in Python." A ramble through several different problems with code of the solvers. The point is to show how easy it is to solve most problems. So easy that a kid could literally do it (part of the talk was about why kids should do it). I missed most off this one because I was hacking [see below] but I'd seen it before so I didn't mind.

Balls

I should read python-dev more regularly. It turns out Hettinger went and implemented fast-C permutations, combinations, and cartesian product in the itertools module. You know, just like the probstat module I wrote. That old code is pretty un-pythonic (I wrote it in my inbetween stage so it is a generic lib with both python and perl wrappings). I had a mostly finished rewrite that was CPython from the ground up and - suprise! - it looks almost identical to Raymond's. Almost, I spun out the iterator into a separate object so the base object could have a len (iterators aren't allowed to support len). His doesn't have random access but that is one of the things no one used on mine so I was going to drop it anyway.

Python Language Summit

2009-03-27T12:25:00.003-04:00

[this is about yesterday, I'm getting caught up]

Yesterday was the Python Language Summit. 40+ of the core developers of all the python implementations (CPython, Jython, IronPython) met to discuss stuff. The meeting was five hours in 1.25 hour chunks. Morning topics included goals for future releases, the timing of future releases, and what processes we need to change, if any. Afternoon topics were how to share more stuff (tests and benchmarks) across implementations, and how to combine different the various setup/packaging projects into one.

Meeting face-to-face is always easier than using a mailing list. The conversations are synchronous, latency is low, and decorum is higher. The meeting could have used a dose of Robert's Rules of Order - it wasn't always clear when we had reached consensus so topics drifted until even tangents had been exhausted. Some quick pronouncements might have been better.

What decisions were made? I'm not really sure. Python 2.7 might be the last in the 2.x series, or it might not. New libraries are more likely to get backported than new core language features. If you were expecting to continue on the 2.x series until a magic day when you found yourself writing 3.x code you will be waiting forever. There was some interest in writing a 3to2 source converter that mirrors the 3to2 tool. Because 3.x has stronger semantics (only one obvious way to do it) a 3to2 tools is theoretically easier to write; but a 3to2 tool might also target many 2.x versions so "easier" is still a lot of work.

The different implementations will be sharing more tests and benchmarks in the future. Probably. There was general agreement we should but it will be up to individuals to make it so (as always). Ditto for packaging - everyone agreed that the different tools should combine but the devil is in the details - so the discussion is being moved to the packaging-SIG list.

In Chicago

2009-03-25T19:17:00.003-04:00

All checked in at the Crown Plaza (where most of the core devs seem to be staying). Took the hotel shuttle over with Glyph and Brian Dorsey. Glyph and I had different flights from BOS to ORD but arrived at the same time. Go figure.

The talk of the town so far is Unladen Swallow - a google effort to replace the CPython byte code machine with LLVM. Goal #1 "Produce a version of Python at least 5x faster than CPython." Holy S**t.

PyCon Tomorrow

2009-03-24T22:35:00.003-04:00

I arrive in the afternoon on the 25th and will be there through April 1st (leaving in the PM).

Tip: DO NOT bring a heavy coat unless you are from a sunshine state. Chicago sounds cold but like last year it is supposed to be warmer in Chicago than Boston (55F+ during the day, 35F+ at night). Last year I stepped off the airplane wearing an overcoat and it turned warm into sweltering hot.

No mustache again this year .. and maybe never again. While it was fun so was being a longhair with an eyebrow ring in the early 90s (dude, it was the early 90s). Many things are fun, once.

I have business cards this year. Plain $0.25/ea Kinkos cards and not the $2.00/pop custom wonders I had at my tech/marketing company. Last year I had neither and felt naked.

Consider the above an announcement that I'm officially back in the job market; tomorrow will be two years exactly that I've been on vacation. My bowling game has improved greatly, my golf game not as much (though to be fair I've been bowling for 2 years and golfing for 25). It was nice to be available to travel to every birthday/baptism/marriage/funeral/etc and go abroad on a whim but it does get old (and the pay stinks). Mainly I'm looking forward to working with a team again. Personal projects are fun but it ain't the same.

Afterward: PyCon on the Charles

2009-03-18T21:54:00.004-04:00

The Boston mini-PyCon went well. The Beta House was at max capacity with 30+ in attendance. I met Jesse Noller for the first time; I'm not sure how this managed to be a first because he's a python dev and has been at the last few PyCons. I'd bet there are pictures on flickr that have both of us in frame.

Noller's talk was a thousand foot view of the plethora of multiprocessing/concurrent/messaging/you-name-it frameworks in python. He compared the current proliferation to the mess of competing web and ORM frameworks of two years ago. Sounds about right.

Taylor's talk was about Reinteract, his spreadsheetish interactive python shell. I expected to sleep through this but the app is actually interesting. It falls somewhere between IPython and Resolver One in functionality. I'm sure he and the Resolver guys will have lots to talk about.

My talk went OK. Half the slides are new (again) and I really like the individual slides, but in the rewrite it lost its narrative (the "Radically Simple" of the title). I loved the idea of jumping in on the second slide with an example titled "Why this is Cool" but it fell flat because the first slide didn't explain the pains and foibles of doing without class decorators. I also need to reinsert the longer explanation of what decorators (and metaclasses and mixins) are. The PyCon crowd will be more savvy than the Cambridge user's group but not that much more savvy. A few people said "that looks really cool but I have no idea WTF you are talking about." My talk is flagged as "advanced" in the program but beginner/intermediate/advanced is ignored by attendees (and usually isn't on the printed schedule).

Bruce Eckel will not be giving a keynote as he broke his leg badly while skiing. This is a mixed blessing for me because Bruce's keynote was about class decorators and metaclasses. Good for me because people won't skip my talk for his keynote. Bad for me because I wanted to see his talk and was looking forward to talking to him. Oh, and now I have a useless T-shirt that says "Bruce Eckel Stole My Talk" on the front and "I Stole Bruce Eckel's Talk" on the back.

Ned Bachelder gave his PyCon talk A Whirlwind Excursion through Python C Extensions at the previous meetup. Do click the link, it includes his slides interspersed with his own commentary. I did a similar talk titled "Writing Your Own Python Types in C" a couple years ago so Ned & I traded notes. One of the slides is a nod to our conversation. It isn't as egoboo as the time time Guido gave me full slide w/ attribution in his "State of Python" address, but I'll take it.

PyCon on the Charles

2009-03-03T10:05:00.006-05:00

Ned Batchelder has organized a preview/practice session of all the Boston based pythoneer's talks. Ned gave his talk last month at the Cambridge meetup, and March 18th there will be a three hour session with Jesse Noller, Owen Taylor, and me at the Beta House. Talks start at 6:30pm. If you haven't been to a PyCon it is a chance to see a mini version of one. I'll even bring the beer.

Talks are:
Noller: Concurrency and Distributed Computing with Python Today
Taylor: Reinteract: a better way to interact with Python
Diederich: Class Decorators: Radically Simple

[fixed] I misattributed Taylor's talk to someone else.

Python Language Summit

2009-03-01T11:49:00.004-05:00

[cross posted from my non-python blog with some minor edits]

I just got my invitation to the Python language summit. I had planned on going anyway, so the invitation just makes it less awkward for everyone involved. The summit overlaps with the tutorial days of PyCon because, well, by definition if you belong in the summit you don't belong in the tutorials [tutorial instructors can suck it].

The summit is interesting because it is unusual* - open source events are usually open access too. The agenda is wide open and is roughly focused on standardization and the future, whatever that is. The invitees are the core developers of all the implementations of Python: regular Python (aka "CPython"), Jython (Java), and IronPython (C#).

I'm not sure what the purpose of the invites is, other than to convey weight. To make the list is to ask "who are the 50 people on the planet who would actually want to come?" Griefers and trolls would be bounced regardless [come to think of it, maybe I was invited because I have no compunctions about bouncing griefers and trolls] so perhaps the invitations are meant to discourage the well meaning but clueless. Every convention has at least a few of those - does anyone remember the "callable None" guy from a few years back? "well meaning but clueless" doesn't begin to describe him.

* It is not without precedent. See the the Reykjavik sprint. [my favorite bit from those posts "Cod jerky smells a bit like feet but tastes OK. Dried shark smells a lot like feet and tastes exactly like dried asshole."]

PSA: Wireless Keyboards

2009-02-13T12:55:00.002-05:00

The batteries in your wireless keyboard don't die at arbitrarily long intervals. You put your cellphone close to the receiver at arbitrary intervals and then move it while changing the batteries.

And by "you" I mean "me."

Look Who's [not] Talking at PyCon

2009-01-22T12:05:00.007-05:00

The schedule is up and it looks good. It is a mix of conference warriors plus some new blood; likewise the talks are a mix of old standards and new topics (some of the new talks are by conference warriors and vice versa).

The familiar names include Brett Cannon (cpython), Jim Baker, (jython), and Michael Foord (ironpython). There is a host of names that might be missing or might not - I can't recall if they do talks every year - but there is no Norwitz, Warsaw, Holden, or Martelli. Noticeably absent is Raymond Hettinger who has given several talks per con at several cons a year. Noticeably present is EVE Online in the form of Richard Tew and Krisjan; with the Icelandic Krona where it is I don't know how they can afford a taxi let alone air fare.

On the new list is Bill Gribble. Bill Gribble gets a free beer for having one of the best names ever (if his Mother shows up she can collect for her work, instead). I bet none of his friends ever start a story "I was having lunch with my friend Bill..." But instead always "I was having lunch with my friend Bill Gribble..." I don't know Gribble from a hole in the wall but I'll make a point of changing that.

[update] Talked to Raymond and he'll be attending, at least (barring life's usual caveats).
[updateder] As Brett and Ivan mention in the comments there is now a tier of invited talks that includes many of the "missing" conference regulars. Glad to have em, the "hall track" is my favorite and many on the invited speakers list make it so.

Post's Machine

2009-01-18T18:38:00.003-05:00

An undergrad CompSci major, Shriphani Palakodety, posted an implementation of Post's Machine. Post's machine is a simple computer that has separate data and execution storage. Here's my implementation and my advice to Shriphani - Keep it simple! When working on hard problems post-graduation you'll find that code gets complicated of its own volition. The job of the writer is to fight it at every turn (the next guy to come along will appreciate the effort).

This version replaces the two custom containers with python dicts and moves all the complicated code to the pretty printer. A collections.defaultdict might be a slightly better choice for the 'infinite tape' that starts as all zeros. A nice thing about defaultdicts is that you can min() and max() even empty dicts.

I also did a python3.0 version which was nearly identical except for a 'with' on the file open and print-as-fucntion. Disappointingly I thought the advanced tuple unpack syntax would help in the case where a too-short tuple is padded and then the padding discarded.


# args might be a two or three tuple
a, b, c = (args + [0])[:3]

# python3 syntax
a, b, c, *ignore = args + [0]

The *ignore argument demands to be read as opposed to the [:3] trimmer on the end which keeps the low profile it deserves.

Here's the code for my Post Machine

def parse(lines):
  program = {}
  for line in lines:
    pos, action, jump = [p.strip() for p in line.split(',') + ['0']][:3]
    program[pos] = action, jump
  return program

def execute(program):
  tape = {}
  tape_pos = 0
  action = None
  action_pos = '0'

  while action != 'exit':
    action, action_pos = program[action_pos]

    pretty_tape(tape, tape_pos)

    if action == '<':
      tape_pos -= 1
    elif action == '>':
      tape_pos += 1
    elif action == 'mark':
      tape[tape_pos] = 1
    elif action == 'unmark':
      tape[tape_pos] = 0

  return tape, tape_pos

def pretty_tape(tape, tape_pos):
  if not tape:
    tape = {0:0}
  min_pos = min(tape_pos, min(tape), 0)
  max_pos = max(tape_pos, max(tape), 4)

  parts = []
  for pos in range(min_pos, max_pos + 1):
    val = tape.get(pos, 0)
    if pos == tape_pos:
      parts.append('[%d]' % val)
    else:
      parts.append('%d' % val)

  print '.....', ', '.join(parts), '.....'

if __name__ == '__main__':
  program = parse(open('post.txt'))
  execute(program)

PyCon Talk Accepted

2008-12-16T09:05:00.002-05:00

I received my talk acceptance email yesterday. On schedule, even! The PyCon organzers get more polished every year (I didn't submit a talk last year so maybe this isn't new). It will be interesting when they publish the stats for talk proposals. I vaguely recall that the rate was 50% last year and 80% the year before that. The attendee numbers doubled between '06 and '07 so the number of proposals likely did too.

The email included the anonymous reviews of the proposal. Most were +1 but I'll keep in mind the +0 and -1s when preparing the new version of the talk. They were mostly concerned that the PyCon UK version of the slides was too code heavy. There is even more material now that decorators are in use "in the wild" so I may shunt those into a separate tutorial-like paper or maybe just include them as slides after the end so people who download the .pdf will see them.

I'll definitely have to prep and practice the new talk well in advance of PyCon. Lots of possibilities there; The Colorado Python group get together to practice all their member's talks every year - we could get the Boston PIG to do something similar. And I could make a show-me-do thing. And I'll definitely give it to the pythoners at my old company as a trial run.

Don't forget to register early for PyCon. We maxed out the hotel some nights last year and this year should be even bigger.

PyCon 2009 Plans

2008-11-12T15:21:00.002-05:00

I'll be there. I submitted a talk proposal for a further refined version of my class decorators talk. Since that talk people have actually started to write useful class decorators that weren't just replacements for metaclass hacks. The total ordering decorator, for instance, modifies a class that implements the __lt__ comparison to have ALL the comparison operators. For the talk I'll also work up a decorator to replace UserDict: define __getitem__ and the deco will add __contains__, etc. I always thought mixins were a hack, frankly.

Post PyCon UK

2008-09-16T17:06:00.007-04:00

Due to my harried transit I didn't get to prepare my lightning talk "PyAsshole: Simulating a partial information non trump card drinking game in Python." I did spend my flight home working on a simulator. It currently onl-y has one strategy (random legal plays) so I have to add a player that uses my preferred strategy [play to win or - when you can't - play not to lose]. I'll also have to implement a drink count in the simulation. The idea is that no matter what the player plays he has a chance of reverting to a random play at a chance based on his drink count.

Unfortunately adding in a drink count immediately injects a social (and thus very hard to model) component to the game. The mechanical rules say you drink if you are skipped or if you pass. But the social rules say you can be made to drink for any reason by someone higher than you in the chain. People being people those optional drinks are subject to retribution and those kinds of things are very hard to model. I expect tit-for-tat to work well in simulated play but that completely ignores the human propensity which is to engage in cooperative play.

PyCon UK: Class Decorators, Radically Simple

2008-09-14T08:28:00.006-04:00

I'm giving a new version of my EuroPython talk, Class Decorators: Radically Simple.

Two thirds of the slides are new. There is much more front end emphasis on what decorators are and simple function decorators examples. The examples of class decorators and metaclasses have also changed.

Post Con

I changed the slide that used the old 3rd party decorator module to use the standard lib functools equivalent.

Speakers thoughts: The details of this talk were substantially different than the EuroPython version. I had more slides but there was less talky-talk involved with each one so it ended up the same at about twenty minutes. I intend to rejigger it again and submit it to PyCon as a mix of the two talks. The more concrete slides at PyCon UK went over well but most of the metaclass banter fell off and all the esoteric stuff (which will probably stay dead). The topical slide about getting my passport was skipped during the presentation - I did it as a lightning talk instead.

I wasn't nervous for the talk itself, there were maybe 75 people in the audience. I was nervous for my 5 minute lightning talk immediately afterward "How to get a new US passport same-day while committing the minimum number of felonies." I don't know why - I've spoken in front of larger crowds before and the idea of lightning talks is that they aren't expected to be polished (I had no slides and just spoke extemporaneously). The lightning talk went well and might have tied for the highest number of laugh-lines with the other humorous talk "Why I love Zeppelins."

Next time I intend to cheat and take some Beta Blockers before any speech. For those who don't know: beta blockers repress the physical symptoms of the fight-or-flight reflex only and don't change your brain. So they will suppress a quiver in your voice or hands but don't help with "umms" or "aahs." The only way to suppress the um/ah reflex is to practice before hand - a mirror works fine; just have enough canned variations on the topic that you can pick and choose the most suitable one at runtime which is more fluid than you think.

I only had one question during the talk and one question afterward. I'll have to think of ways to introduce more uncertainly into the talk for my next version. The first version At EuroPython elicited two or three questions during the talk and two or three after. Based on that feedback I answered those in this version but I'd like to generate some new ones next time.

PyCon UK Flight Plans

2008-09-10T21:07:00.004-04:00

What to do with on a 12 hour flight between Boston and Birmingham (1 stop in Amersterdam)?

I could work on my presentation. This would be OK but not ideal since I don't have internet conectivity to look stuff up.

I could work on my lightning talk: "PyAsshole: Simulating a partial information non trump card drinking game in Python." I'll do some of that but I only have five hours of batteries. [if you aren't familiar see here for one set of rules]

I could bring Neal Stephenson's Anathem but I could fit two or three other books in my bag instead of that door stop; I'll do that because if I can't get into his book I'm stuck with it.

I'll likely do a mix of PyAsshole and books with at least one of those being Heinlein's The Moon is a Harsh Mistress which is frequently cited but which I've never read.

Speaking at UK Python

2008-08-11T01:29:00.002-04:00

The talk schedule for UK python is posted and I'm on it. Conferences are a good time by themselves but I'd be lying if I said I don't enjoy talking in addition to attending.

The schedule is a bit funky with four rooms instead of the standard three. It is an interesting experiment so I hope one of the organizers will write up the reasons for, and the results of afterwords. UK Python is expected to be small at ~200 people (the size of PyCon 10 years ago). Small ventures are expected to experiment because the cost of failure is low and the rewards of success are high. I don't have a horse in this race so I'm happy to watch and see.

UK Python Plans

2008-08-08T20:23:00.004-04:00

I submitted a talk this week which is hopefully not too late. I do like the idea of rolling admissions as opposed to the aspirational deadlines of other Py* conferences. "The dealine is Monday! and this time we really mean it!" The talk is a version of my EuroPython one with feedback from the first time I gave it.

I'll be there for UK python even if my talk isn't accepted (please disregard this if you are on the UK program committee). I've technically been on English soil before but practically I don't think airports really count. By the same token I've visited California and Hawaii.

See you there and hopefully the yobs, chavs, and hoodies aren't as bad as the UK papers make out*.

* There were many commonwealth attendees at EuroPython and the pan-national conference attracted ex-patriots in particular. I've always toyed with the idea of being an expat and some nudged me to expatriate to the UK. I dismissed the idea with some bluster about not being able to carry a gun or a proper knife. Not that I carry a gun; the nice thing about being an American is the free rider effect (crims don't know that you aren't carrying, they just know that some people are). I will be at the range tomorrow and will likely post pictures on my non-python blog.

German, American, or Other? Part II

2008-07-28T00:39:00.005-04:00

My PyCon 2008 posts included an item "German, American, or Other" with a list of core python developers (anyone with subversion privs) and a guessing game on nationality. I just updated the post with some statistics on how many Americans come from German immigrants (hint: lots) but I'll repeat the update up top.

German ancestry is dominant** in 17% of Americans or about 50 million people. You can be 1% German and have a German last name or 99% German and have a Spanish last name but if people mate randomly* the number of people with German last names would be at least as high as the number of people who self-report that they are mostly of German ancestry. There are 80 million Germans and somewhere near that number of Americans with German last names so this quiz becomes somewhat less surprising in hindsight.

* and mate randomly they do. The urge to mate is much stronger than any weak favoritism for ethnic continuity. In the US the idea that like-marries-like doesn't last for two generations (see "My Big Fat Greek Wedding"). And the idea is weakest in the majority (aka white people). Growing up I was amazed that the O'Dea kids down the street were 100% Irish. They weren't recent immigrants so the only thing in my mind was: wow, what are the odds?

If you want another anecdote, here's one: my father's father (who's last name I have, obviously) did not like the Pennsylvania Dutch ("Dutch" being the popular corruption of "Deutsch") . He was a 3rd generation New York German but he didn't care for the country bumpkin Pennsylvania Germans who got on a Westward boat 10+ generations before his own ancestors. His bias was entirely theoretical and limited to jokes and stereotypes. He didn't consider my mother (100% Dutchy) or any of her family as "Dutchy" because they were doctors, lawyers, businessmen, and engineers. He was encouraged to not bring it up and everyone else was reminded that he didn't really mean it. "Etlay itway opdray" is our family motto so this was rarely a problem.

** German ancestry is a plurality but thankfully English/Scottish law is dominant. At the time of the Constitutional Convention in Philadelphia, Pennsylvania the dominant language in Pennsylvania was actually German. Those German speaking Americans were not parochial, however, they voted to make English the official language of the state.

NB, if you ever read a story about harsh race relations in America please take my grandfather's story into account. My grandfather's rhetoric far exceeded reality. His words (accurately quoted!) would have been an indictment but were far different than his beliefs in practice.

QOTW

2008-07-22T18:23:00.003-04:00

I suspect you've spent too much time using Python when you start calling C variable assignments "mutations" ;-)

Fredrik Lundh in a thread at Ned Batchelder's blog.

ICFP at a Glance

2008-07-11T19:00:00.009-04:00

This years problem is simple and very very hard (otherwise it wouldn't be much of a contest). It is a hard path finding problem with time constraints like many past years. Unlike recent years the problem doesn't favor big teams or teams with lots of CPU power. OK, it favors them a little but not much.

The problem is to write a "Mars Rover" control program that communicates over a TCP/IP socket and receives and sends messages with a small real-ish time delay (around ~100ms). All submitted programs will run on the same hardware so the team with the best algos will likely win.

Bob is out but I'm still sticking to the bourbon and steaks theme. I spent an hour on the porch reading and interpreting the problem. And I brought some friends:

Beside Huey, Duey, and Louey are the printed problem description, a fifth of Bulleit bourbon, my engineering notebook, and a trusty Uni-Ball Roller (blue).

If it doesn't violate the rules ... here is how I'm approaching the problem for a first draft:
* ignore the delay in sending commands
* treat all boulders and craters as square
* treat hostile martians as immobile boulders

As the joke goes a Physicist was asked to guess how much a horse weighed and said "Well first we assume the horse is a sphere..." Good enough for a first estimate.

You will notice that I'm a good sport - the rubber bands have been cut on the lobsters. Also, lobsters aren't a big deal in Boston. They cost $8/lb (similar to luncheon ham) so what would be extravagant in land-locked Iowa costs just $30 here.

The problem has a bunch of complicating quirks, of course. It may be profitable to crash into a boulder to slow down and on some maps it may be best to suicide as quickly as possible because of the scoring system. 20 grad students spent a month designing the problem so I'm sure more subtleties will out. Bastards.

Aside I should have mentioned that I went bowling before food shopping and the problem was published. 152, 159 - not good but not terrible considering I was jet lagged and I was a week out of practice. Plus the lanes were messy (that's my excuse and I'm sticking to it).

ICFP 2008 Starts NOW

2008-07-11T11:11:00.006-04:00

The ICFP programming contest starts in a couple hours. I have done it the past five+ years and care enough that I flew back from EuroPython early just so I could participate. Unfortunately my usual compatriot Bob (an Electrical Engineer) has been sidelined by an illness. This is a double blow because not only do I not have a second, our normal weekend of beers and grilling (really $50 bottles of bourbon and Omaha Steaks) is off.

I'll be doing it anyway but with a reduced level of fun (and likely success). If anyone reading this wants to participate let me know. I'll be running a bzr repository and depending on the problem (which changes by year) the code will be somewhere between 90% python and 90% C.

Foord, Brandl? I'm looking your way.

I've been meaning to publish my contact information for a long time; John Patrick has been a big deal at IBM for a long time and published "Net Attitude." He makes all his contact information (phone, IM handle, etc) publicly available and swears that is no more than a nuisance. Previously my information was unpublished and the biggest nuisances were head hunters and telemarketers. To put it a different way: all the annoying people already can find me, so why not make it public?

So if you are looking for semi-serious team for the ICFP contact me at jackdied@gmail.com or 617 821 1734 ('mercian). If you're in Boston there are a few wifi enabled cafes/bars/restaurants where we could collaborate.

3 hours until the problem is published.

EuroPython Errata

2008-07-10T23:41:00.005-04:00

Hettinger used tinyurl URLs in all his presentations to give short URLs to resources like CPython source code. Unfortunately I miss-typed a '3' into an 'E' and landed on a gay porn page. This wouldn't have been a problem but I was sitting in the front row and the room lights were out so my screen was BRIGHT. I couldn't just close the page because it had popups, I had to bookmark it first, etc.

The price of a con varies but costs about the same as a vacation. For EuroPython the plane ticket from Boston to Vilnius was $1400 ($600 for the airline, $800 in taxes!), conference hotel was 100 euros/night, walking-around money was $60/day. If you plan ahead (which I did not) you can save money on everything but the airfare. Some of the attendees stayed at a hostel (15 Euros/night) or rented an apartment (50 euros/night). Food and drink prices vary by city: PyCon Chicago was about $100/day, and the Iceland sprint was about $200/day. You can avoid that by buying food and drink at a supermarket which are reachable by public transport. Conventions are vacations for me so I voluntarily spend quite a bit around town.

Vilnius was unusually cheap. The public bus from the airport cost 1.10 litas (about $0.30 US). I thought the hotel was unusually cheap - the venison (deer) plate was 35 litas ($11 us) and 0.5 liters of beer 9 litas ($3 us). The hotel was expensive - prices in town were around 25 litas for a meal and 4.5 litas for a beer.

Conferences should have water and coffee available ALL THE TIME. The PyCon organizers tried to do this but the hotel staff sometimes cleared the service. At EuroPython coffee was available all day but water only rarely; the hotel had no water fountains, none at all (are water fountains an American thing?).

Vilnius is not a wealthy country. Our hotel seemed to be the center of the "jet set" for Lithuania. The casino in the hotel basement was full of locals spending flashy money and the stip club across the street the same (so I am told).

Bars close whenever they feel like, as do restaurants. We had to leave a diner/breakfast place at 7am because they were closing. The casino was open 24 hours a day but to get in you must give them all your information and they take your picture. I think I mostly thwarted them with my state drivers license and a bad picture. I didn't gamble - the rest of the hotel barstaff went on strike but the casino did not.

A Vilnius taxi ("taksi") costs twice as much if you hail one on the street instead of calling. This is the local law. So if you need a cab have the hotel call one for you.

I was worried about attendees English (mostly for my presentation). The ESL (English as a Second Language) English was as good at EuroPython as at PyCon. Also, the material being technical python stuff most of the written content is understandable even if you don't speak the language (I can read Brandl's PyPy talk just fine and I know zero German).

PyCon is about 3/4 American and 1/4 other. EuroPython was the reverse.

If you are standing in a group of people and two of them have speaker's badges then ALL of them will have speaker's badges. This isn't because of elitism; it is because the speakers tend to speak everywhere so they already know each other.

Class Decorators: Radically Simple

2008-07-09T08:34:00.006-04:00

The slides for my presenation are now online: Class Decorators: Radically Simple. Hopefully the pdf export worked OK.

The talk went reasonably well but I went a bit too fast and finished in 20 minutes instead of 25. I did have a full room.

Update Christ on a crutch and other blasphemies - The talks weren't recorded but I turned on my video camera and left it next to me on the desk. The audio was pretty good and oh boy - I really talked much too fast. I need to master talking in front of 150 people at the same speed that I talk to five.

I also dislike what my voice sounds like on tape, but that is a normal thing. Also, most people can't expect to be as sonorous ss Alex Martelli (an Italian with a booming voice and a English diction that would put most teachers to shame).

Off to EuroPython

2008-07-05T13:49:00.005-04:00

My flight out of Boston leaves in a few hours. I'll be arriving in Vilnius on the 6th at 2pm (Lufthansa 3252). If you see me in the Airport I'll be the tall redhead with a Lithuanian phrasebook in one hand, looking all American-y.

(short hair and no mustache again, like PyCon2007 and not like PyCon2008)

later: "looking all American-y" didn't work out. The Lufthansa stewardesses spoke English to the English, French to the French, but German to me. Sure, I am more than half genetically German (and I look it) and .. I didn't exactly dissuade them by answering "coffee, ja" when the drinks cart rolled around, after she had said something unintelligible to me that must have been "something to drink?"

See You at EuroPython

2008-06-08T00:03:00.003-04:00

My talk was accepted for EuroPython so I'll be invading Lithuania July 7th-12th (probably arriving on the 6th and leaving on the 13th to blunt the jet lag).

Unfortunately this conflicts with the 2008 ICFP contest on the 11th-14th. In past years (2007 writeup) I've done the ICFP as a long weekend steak and beers hack-a-thon with friends. Maybe we can get a BoF going at EuroPython.

Doh, Lebowskifest is that weekend too.

Boston Python Life

2008-05-21T16:58:00.002-04:00

I still haven't been to a Cambridge python meetup in a long time and I just blocked Wednesdays again by signing up for a bowling league that night. A man has to have his priorities.

Will G's Nomadic Computing Herd now has a schedule. I've worked with Will on open source projects in the past and lived five minutes away from him for years but we've never met. Doug Napoleone quipped at PyCon "I travel 1000 miles to see people that live next door." Exactly (Doug also lives five minutes away and I hadn't seen him since last PyCon).

EuroPython, Maybe

2008-05-21T16:33:00.002-04:00

I submitted a talk proposal for EuroPython. It is a longer version of the lightning talk I didn't give at PyCon (slots filled up really super early).

If it is accepted I'll be in Lithuania. I would like to go regardless but PyCon cost 2k+ and because the Fed has been printing dollars for the last 20 years I suspect a trip to Europe would cost even more than that.

Shell Count Meme

2008-04-12T20:36:00.003-04:00

history | awk '{{a[$2]++) END(for(i in a) {printf "%5d\t%s\n", a[i], i))' | sort -rn head

awk: line 1: extra ')'
awk: line 1: missing ( near END
awk: line 1: extra ')'
awk: line 2: missing } near end of file

Take that!
(actually I use tcsh which doesn't keep history between invocations. So this output is as interesting as it gets)

Misc from PyCon II

2008-03-23T00:18:00.009-04:00

I wasn't active on the Pycon-organizers list this year but this game suggestion did get some laughs at the python-dev core sprint. So I give you:

German, American, or Other?

(All of these guys have svn commit privs on the core, as listed here)

Hettinger: American
Brandl: German
Lundh: Other (Swedish)
Reifschneider: American
Doerwald: German
Diederich: American (my great^8-grandfather was German)
Martelli: American/Other (originally Italian)
Dalke: American
Kuchling: Other (Canadian)
Ronacher: German
Warsaw: American
Janssen: American
Heimes: German
Tismer: German
Goodger: Other (Canadian)
Wolever: Other (Canadian or just in Canada?)
Forsberg: Other (Swedish)
Yoshida: I have no idea but I'm hoping this is a trick question
Stein: American
Niemeyer: Other (Brazilian)
Seutter: Other (Canadian)
Kaiser: German (?)
Gustaebel: German
Lemburg: German
Loewis: German
Klose: German
Norwitz: American
Schemenauer: Other (Canadian)
Oussoren: Other
Heller: German
Wouters: Other (Dutch)
Seilnacht: Other
Petursson: American (probably, he speaks fluent Icelandic but I think he converted as a youth)

[highlight the text after the ':' to see the answer]

No doubt there are some errors in there so please leave corrections in the comments. Google isn't great for determining nationality and I didn't even google the answers I already "knew." Also, apologies to any Austrians who got lumped in with the Germans (a friend's grandmother at the age of 90 told her son she was Austrian after saying she was German for a lifetime. When confronted she replied "German, Austrian, whatever.")

Update (July 27, 2008) German ancestry is dominant in 17% of Americans or about 50 million people. You can be 1% German and have a German last name or 99% German and have a Spanish last name but if people mate randomly* the number of people with German last names would be at least as high as the number of people who self-report that they are mostly of German ancestry. There are 80 million Germans and somewhere near that number of Americans with German last names so this quiz becomes somewhat less surprising in hindsight.

* and mate randomly they do. The urge to mate is much stronger than any weak favoritism for ethnic continuity. In the US the idea that like-marries-like doesn't last for two generations (see "My Big Fat Greek Wedding"). And the idea is weakest in the majority (aka white people). Growing up I was amazed that the O'Dea kids down the street were 100% Irish. They weren't recent immigrants so the only thing in my mind was: wow, what are the odds?

If you want another anecdote, here's one: my father's father (who's last name I have, obviously) did not like the Pennsylvania Dutch ("Dutch" being the popular corruption of "Deutsch") . He was a 3rd generation New York German but he didn't care for the country bumpkin Pennsylvania Germans who got on a Westward boat 10+ generations before his own ancestors. His bias was entirely theoretical and limited to jokes and stereotypes. He didn't consider my mother (100% Dutchy) or any of her family as "Dutchy" because they were doctors, lawyers, businessmen, and engineers. He was encouraged to not bring it up and everyone else was reminded that he didn't really mean it. "Etlay itway opdray" is our family motto so this was rarely a problem.

NB, if you ever read a story about harsh race relations in America please take the last paragraph into account. My grandfather's rhetoric far exceeded reality. His words (accurately quoted!) would have been an indictment but were far different than his beliefs in practice.

Bonus PyCon Day!

2008-03-20T18:39:00.002-04:00

I was supposed to fly back to Boston tonight after a successful PyCon but the fates intervened. Or United Airlines sucks - take your pick.

Well, United isn't entirely useless because they did manage to get my bags to Boston. Bastards.

I do get to sprint for an extra day and I was able to get a room in the conference hotel. Joe, the all knowing bartender, says that the airlines booked a block of 250 rooms here for tonight. It should be a pretty ecclectic and angry crowd at the bar.

Update: I made it home the next day (Friday) but last I saw David Goodger his flight was canceled and he was expecting to get out Saturday. From the customer service lines at the airport David probably had some company.

NB, perhaps my mistake was traveling without a towel. That's like asking for trouble.

Misc from PyCon I

2008-03-18T18:05:00.002-04:00

Don't stay at the Double Tree (the hotel next to the conference hotel) and if you own their stock sell now. I was impressed when checking in because they gave me a hot cookie (that is not a euphamism). I asked about it and they pointed out the OVEN behind the counter. I thought these people really have their shit together - even thouogh I didn't want a cookie. But the clocks in every room were wrong by an hour. Worse, the clocks are set by a radio signal and have no way to manually set the time. The hotel knows about the problem but isn't trying to fix it. That hour is because congress, in its infinite wisdom, wanted to save us all money at no expense to themselves by screwing with Daylight Savings. Congrats congress: you have cost three hundred million people five minutes of inconvenience a week for a year and counting.

Do stay at the Crown Plaza (the conf hotel). It has some rough spots - the bar doesn't open until 4pm - but has fewer signs of dysfunction than the Double Tree. I left a book in a common area yesterday and today asked a waitress if they had found it and saved it. She told me they had found one book but asked me to describe it instead of showing it to me. It turns out I had used a $20 bill as a bookmark and they wanted to make sure they had the right guy first. Actually I had two book marks in there and the other one was a plane ticket stub. Another employee had left a message for me at the desk. So yeah, the Crown Plaza franchise might be average but the local staff are honest and hard working.

PyCon Core Sprints

2008-03-17T12:25:00.002-04:00

Sprints have started. The core python sprint has 30 guys this year which is huge (about 12 are core committers). The Django sprint is maybe twice that size. Thankfully no one has revoked my privs since last year even though I haven't committed anything in ten months.

First I'll be investigating PEP 3115 (Metaclasses in 3k). I tried some of the syntax yesterday and couldn't get it to work so either the implementation or the PEP is off. It's probably the PEP in which case my first checkin will be a doc fix.

PyCon is Over

2008-03-16T17:14:00.002-04:00

Well, the main conference anywawy. I'll be around for sprints.

I prepared a lightening talk but didn't get to do it (the talks were a bit disorganized and the slots were full instantly). Mine would have been on class decorators in python. Originally it was titled "in python 3000" but someone back ported decos to 2.6 recently. A pdf of the slides is available here. Most of the content was to be spoken so the slides are sparse.

See You at PyCon

2008-03-11T10:52:00.003-04:00

I thought I procrastinated last year -- but a friend just asked me what I was doing for St Patrick's day and I said *!#@(*%. The good news is that this morning I was able to book a hotel room and a flight. The bad news is that I couldn't get the conference rate at the hotel and I have to stay next door on the 13th. Live and not learn, I always say.

If you are looking for me I look pretty much the same as I always have, but mustachier.

Holy PyCon!

2008-02-19T15:29:00.002-05:00

I just tried to register for PyCon (and failed, the fraud detection hates me apparently). The shocker is that I was reg number 585 and the early bird period hasn't even ended! Undoubtedly some of those were test registrations but still - this should be a much bigger PyCon than the previous one which was also record setting.

See you in Chicago! (I'll be staying for sprints too)

Boston Python Life

2008-01-20T02:43:00.000-05:00

My old habit of missing Boston python meetups for no good reason is over. I'm now in a Wednesday bowling league so I have a good reason.
Or a reason, anyway.

The Wingware hosted Boston-PIG list seems dead. There was some shuffle when the Boston group became the Cambridge group. I think there is a Plone group too which is sometimes the same people. Maybe Doug could clear that up.

Will Guaraldi's Nomadic Telecommuting Herd occasionally meets across the street from me. Maybe I'll run into them sometime (but they don't have a schedule, so that's hard to do).

In The Beginning

2008-01-14T07:18:00.000-05:00

This is the story of how I started using python.

Out of college I was a well indoctrinated C++ guy. My resume had silly goals on it like "desires to work at a CMM level 3 or higher organization." It was the mid 90s so I ended up working at a C++ based dot com instead (Musicblvd and later CDNow). I left two years later with my youthful enthusiasm for the "correct way" way stripped. It was now the late 90s and my next job was working at a Perl based dot com in Boston (OpenAir). Things were just so much easier with Perl. And everything was different: OpenAir didn't have a single machine that cost over $5k whereas Musicblvd had more than one that cost over $1Mil (a fully loaded E10k ran the database*). Release cycles were monthly instead of annually. Everything was so plainly better and more productive it wasn't even funny. I was no longer a C++ guy.

So when I quit during the bust in 2001 to start my own company I wanted to use perl. I was afraid to for legal reasons so I started casting around for another "P" language. We did the prototype in PHP. The prototype worked but the code was ugly and slow. Python had some buzz so I downloaded that and translated our small app from PHP to Python. The resulting code was a third smaller, easier to read, and ran faster. If that wasn't enough the C guy in me loved the Python core. Compared to Perl and PHP the Python core was simple, clean, and without much baggage. I was sold.

It is many years later and I'm still happy with Python. I have even added one new feature to python (class decorators) and of course, one bug too.

* and that 256 processor Sun wasn't infallible. When it crashed (which it did) Sun helicoptered in tech support and basically paid us off not to talk about it.

Three Flavors of mod_*

2008-01-13T15:51:00.000-05:00

Ian Bicking talks about the conceptual flavors of web deployment. Go read that because this post is only tangential to his point.

Under the Apache 1.x series I have used mod_perl, mod_php, and mod_python. All three do things slightly differently with regards to loading code. Keep in mind apache 1.x only uses the pre-forking model so each request handling process starts life as an exact copy of the root process. The root process's main job is to spawn child processes. [everything below was true in 2000 and for apache 1.x. Don't make decisions based on my recolections unless you plan on running some ancient servers.]

mod_perl allows you to preload scripts when the root process starts. This means no matter how large your code base you only have to compile and execute it once and then every child process starts life with the same initialized modules already in place. At a perl-based company where I worked the preload script was a single line

use Everything;The 'Everything' package loaded, well, everything (is this a standard perl-ism nowadays?). Everything.pm was the most delicate module in a large code base. It imported all the packages in the code base in a way that avoided cycles. As a downside the server couldn't use apache's "graceful" restart because the root server didn't have a way to unload the perl runtime and reload Everything. Besides reloading everything was slooow on the 400MHz CPUs of the day. The work-around was to have a proxy server that could point to one of two local ports that ran mod_perl. To gracefully switch the unused app server was stopped and restarted and the proxy was repointed to the new local port.

mod_php is a bit of yuck by comparison. Or a joy for the reasons Bicking states. The yuck is that mod_php compiles and executes scripts anew on every request. This is great for shared hosts as it acts like CGI but with less overhead because it doesn't have to fork on every request. This sucks for application servers that know they want to run the same code over and over again. It gets worse when the codebase is large and there is a package like Everything.pm that loads the entire codebase of a good sized application on every request [yes, this post is somewhat autobiographical].

mod_python is somewhere between mod_perl and mod_php. The first time a child handles a request it pays the price of loading and executing a script. After that all requests handled by the child are quick because sys.modules still contains all the modules loaded by previous runs. mod_python could support preloading modules in the parent process like mod_perl with a trivial patch [see the mod_python archives circa 2001 for the patch I submitted] but it was decided that the security risk (because the parent process usually starts as root) outweighed the benefit of child nodes having a longer startup time.

In conclusion I would urge you to read the source of the mod_whatever for the language you use. They are small and all work give-or-take the same. It also gives you some idea of how the internals of your favorite language work. My first trip was into mod_perl and I found it readable but with a lot of baggage. When I read mod_php I thought "what a hack! Is this broken on purpose?" Finally I read mod_python and with my perl background I was aghast at how simple and clean the interpreter related code was.

See you at PyCon!

The AWK Programming Language

2007-12-21T11:23:00.000-05:00

On Raymond Hettinger's advice I picked up a copy of "The AWK Programming Language." At 200 pages it makes "The C Programming Language" look like a door stop by comparison. I'm told Chapter 2 is especially good. Chapter 2 is 50 pages out of that 200.

Up til now I've been modeling my writing on Stevens and specifically "Advanced Programming in the UNIX Environment." That book truly is a door stop and I've consulted it too many times to count. Need to write a quick and dirty TCP/IP server? It's in there. Can't remember how to properly fork a daemon? It's in there (don't forget to fork twice!) It is an incredible achievement for a computer book to still be useful over 15 years after it was published.

Raymond also says he has checked in a bunch of optimizations. I'll have to 'svn up' and maybe write about them here (I say that just to increase the likelihood of doing it).

Python stuff goes here

2007-12-07T20:33:00.000-05:00

The python section of my main blog (over here) was dying from neglect so I'm starting one here. I won't be porting the old content because there wasn't much of it and it was time based besides (updates from PyCon, etc).

My Python TODO list is relatively unchanged:
* Book
* Rewrite the probstat extension
(permutation/combination/cartesian of iterables)
* Get active in python development again

I didn't put in a talk proposal for PyCon but I will be there. I will be preparing a couple lightning talks to test out ideas for full talks for PyCon 2009.