While attending FOSS.IN, one of the sessions was a website security session. I only caught the tail end, but it felt very similar to a session at OSCON this year by Nitesh. I also attended part of Rasmus' talk where he talked about a tool he uses for detecting XSS. Which in turn motivated (reminded) me to examine some of the python tools for programming and simulating web browsers. I checked out both Mechanize and Twill by writing scripts that convert TestGen4Web output to a python unittest and twill script respectively.
Both twill and mechanize seem to work pretty well and appear to be actively developed. I had some issues with twill saying "& was unepected at this time" (still haven't resolved that, though repeating the command doesn't cause a repeat of the error...) while running scripts. Twill appears to nicely ignore robots.txt (and possibly update the user agent?), whereas I needed to explicitly tell mechanize to both ignore robots.txt (as well as change the user agent to impersonate moz). That behavior seems reasonable since twill is meant to be interactive.