Friday, April 3, 2009

Fixing telnetlib

During the PyCon sprints I re-assigned all open (and unclaimed) telnetlib bugs to myself. The biggest longstanding complaint about telnetlib is that non-trivial negotations aren't possible because the negotiation callback is very bare bones. The biggest problem with telnetlib is that there is almost no test suite - which is why some bugs have been open for seven years. So my priorities are first to test the hell out of telnetlib and second to improve negotiation.

The negotiation problem is clearest when dealing with two-way communications like NAWS (Negotiate About Window Size). The first time the server asks DO NAWS the client can reply WILL NAWS and include its current window size. The current negotiation callback supports this just fine. But when the client resizes its window it needs to be able to tell the server, which means Telnet needs a hook for a pending negotiations queue. And forget about the STATUS code which asks the other end of the connection to say what options it thinks have been negotiated - the current Telnet has no notion of state.

Below are the raw TODO and research notes I put together in a few hours at sprints. I used google code search to find some of the attempts to fix telnetlib by either subclassing it or writing a semi-compatible Telnet-alike from scratch (these are harder to grep for, for obvious reasons). The RFCs section marks each RFC as Must/Will/Won't implement. "Must implement" means core stuff for the Telnet class, "Will implement" means the telnetlib should include a negotation implementation for that RFC, and "Won't implement" means it won't (because the RFC is either archaic or otherwise unused in the wild). The BUGS list includes all open bugs and the closed bugs I want to revisit or double-check.

* Testing
- test the read_* gaurantees
- test timeouts (already implemented?)
- test the sb handling
* make real negotation possible
* add real timeout and prompt exceptions
* make Telnet objects context managers
* process_rawq is a train wreck. Make sure we do something compatible but less icky.
* figure out where the hell they found all those contstants.
* Why is chr(17)/"\021" blindly filtered out of the stream?

---- BUGS ----

telnetlib process_rawq buffer handling is confused
SO_REUSEADDR doesn't have the same semantics on Windows as on Unix
telnetlib expect() and read_until() do not time out properly
Issue with telnetlib read_until not timing out
Solaris: EINTR exception in select/socket calls in telnetlib
TelnetPopen3, TelnetBase, Expect split
[THIS, a rewrite of telnetlib. Mine for good stuff]
improve telnetlib.Telnet so option negotiation becomes easie
chr(128) in u'only ascii' -> TypeError with misleading msg
telnetlib.Telnet does not process DATA MARK (DM)
Telnetlib dosn't accept u'only ascii'

No way to disable socket timeouts in httplib, etc.
Telnet.read_until() timeout parameter misleading don't block on IAC and enhancement
ability to pass a timeout to underlying socket change to ease option handling.
telnetlib option subnegotiation fix
terminal type option subnegotiation in telnetlib

---- RFCs ----
Wikipedia lists all the relevant RFCs at the bottom.

Short Description
Will/Won't implement
List of officially assigned option codes
Must implement.
(1983) Telnet protocol definition.
Must implement.
(1983) Telnet negotation.
Must implement.
(1983) Telnet binary protocol.
Won't implement. This was obviated by Kermit, Zmodem, and the like.
(1983) Telnet ECHO negotiation.
Will implement.
(1983) Supress Go-Ahead. Nego supression of "your turn" messages for full duplex connections.
Won't implement. (Obsoletes
(1983) Telnet status. Ask other party to retransmit what they think the current negotiated options are.
Will implement.
(1983) Timing mark. A work around for servers that can't read the socket as fast as people type (!!!).
Won't implement.
(1983) negotiating about negotiating
Proln't, Doubtful this is still in effect.
(1983) End-of-Record code.
Might, I have a vague recollecting that this is used as a prompt sigil.
(1988) NAWS (Negotiate About Window Size)
Will implement.
(1988) Baud rate negotiation
Won't implement. (Obsoletes
(1989) Terminal type negotiation
Will implement. (Obsoletes
(1990) Telnet linemode nego. Basically save packets by being less interactive.
Won't implement. (Obsoletes
(1992) Terminal flow control. Local terminal stuff.
Won't implement.
(1997) SLIP-lite protocol for sharing a modem.
Won't implement.
(2000) Telnet Encryption nego.
Won't implement (does anyone actually use this?)
(2006) IBM iSeries hardware telnet extensions.
Won't implement (starngely, the RFC argues against implementing itself)

---- Alternate Implementations ----
[found using google code search]
a hacky ECHO negotiator

subclass-and-patch NAWS negotiator

a from-scratch wrapper

a from-scratch reimplementation w/ better (but unpythonic) negotiating.


Samuel said...

I forked telnetlib once and improved the negotiation a bit (forcing the remote site to use a dumber terminal type to avoid some control characters).

In addition, I created a wrapper around it to create a common API around Telnet and SSH objects. This should be standard these days.

Also, providing hooks into the data stream (on_receive and on_send callbacks) was something notably absent from telnetlib, which kills it for many applications. (This is also addressed in my telnetlib fork/hack.)

The Telnet/SSH module that wraps the adapters is here:

The telnetlib fork is included with the package and can also be seen here:

Steve said...

Jack: thanks for giving a much appreciated but often neglected part of the standard library some love.

Marius Gedminas said...

Blog posts like this are very inspirational even when one doesn't particularly care for the subject (telnet? was that the name of that obsolete encryption-less version of SSH? ;-).

Add 100 geek points to your score!

rtalexander said...

Unfortunately, there are a number of devices that use telnet to transfer binary data, e.g., via ymodem. There are a slew of such devices in the electric power industry.