I've blogged about bugs and testing in open source previously, I was quite interested in this panel. I was surprised because there was little discussion of code coverage, because I think it is quite important for dynamic languages to have good coverage. (I find that doctest and coverage.py not working together is a huge warning sign that people are ignoring coverage). Anyway regardless of my rant, here's a summary of the testing panel. Moderated by Grig Gheorghiu of Testing tools wiki
- Benji York: zope.testbrowser
- Brian Dorsey: py.test (represent Holger Krekel)
- Chad Whitacre: testosterone (created) switched to Nose
- Ian Bicking: paste.test.fixture, minimock, FitLoader
- Jeff Younker: PyMock
- Kumar McMillan: fixture - module for loading and referencing test data
- Martin Taylor: test framework within TI
- Neal Norwitz: PyChecker
- Tim Couper: WATSUP (windows GUI Testing)
- Titus Brown: twill, scotch, figleaf, pinocchio
Intros
BY: testbrowser not currently Zope-centric. Useful for functional tests. Built on top of mechanize. Important that you can read the test and tell what it is doing.
BD: py.test, minimal api. Goal: no api, just use assert. py.test now has a release, previously only svn possible. Automatic test discovery discovers, full doctest.
TC: Absence of windows testing tools. Make tests easy to run from a gui.
MT: TI has silos of test automation. Try to standardize on a testing language using python. Replace Testdirector with python api to talk to (Testopia, testlink... or one he hopes to write). Has approval to release as open source. Hoping to release some time this year.
KM: Building an ETL system in python. Made a module called "fixture" which loads in data sets at the beginning of tests. Colleague wrote "nose" (Jason Pellerin).
IB: Was interested in Fitnesse testing. Write fitloader, then decided that fitnesse was stupid too. Minimock was a test to see how small he could get it. I hate unittest and want it to leave the stdlib.
NN: Testing should be done to prevent bugs. Pychecker tries to find bugs. Work at Google is to promote testing internally.
JY: Mock objects let you separate your units of code from the rest of the program. Not easy to mock out results of generators. Still a lot of work to do with it, error checking, reporting, playback utilities.
CW: Testosterone is manly testing interface. I'm addicting to dots. Attempt to put some sanity on top of it.
TB: Twill is similar to zope.testbrowser and paste.fixture. Built on mechanize. Useful for testing and monitoring websites. Doesn't have zope or wsgi in the name anywhere.
How do you approach testing in your own projects? Unit, functional, GUI? What provides the most payoff?
BY: Write tests and prose first (documentation driven design). One day the bullet whizzed passed my head, and I've been addicted since them.
BD: Unittesting. Specifically test first.
TC: Depends on where you are in the organization.
MT: Most effective is API testing. Above unit testing. Around 2 million testcase.
CW: That's A LOT OF DOTS. (laughs)
KM: TDD made sense at first but made my head hurt. Makes sense to write functional level test first. Some of our code doesn't have unittests because we test at a higher level.
IB: I've sometimes tested too much. Lots of low level, and very few high level. I hate that you can loop over strings. (Don't know if it's a list sometimes)
NN: Lots of unittests like Ian. Need some level of system tests.
JY: Don't do much function or integration testing at my work.
CW: Tried TDD but didn't convert. Testing mindset different than design and coding. I enjoy more being in the creative design mindset. Will get code out first but later put in unittests to satisfy the dot addiction.
TB: Don't agree with Chad. Kevin Kelley, Fast, Cheap and out of control. Describes coding and testing style. I don'd do TDD, I do Stupidedly driven testing. Just test.
IB: I wanted to test for a long time and hated the tools. Once I found (built) the tools, I was happy (and enabled to test).
Confessional: Is there a recnet example when you didn't live up to your testing standards?
TB: I don't have standards.
CW: More important to code and get your ideas out than test.
JY: All the time I don't meet my standards, then the code slaps me hard and I have to go back.
NN: Dashboard at Google for test results. Only unit tests were written no system level. And that's sufficient. Be pragmatic and do what you need.
IB: I don't test well when I don't understand the problem I'm trying to solve. Really need to write it once, test and then re-write if you don't understand problem.
KM: Most important thing is having a regression test suite. You know when you break something.
MT: Confession, I don't write enough tests for my test framework
TC: Haven't talked about manual testing much, but it's very useful.
BY: If you are prototyping you probably don't need to test. When I'm in "hack" mode, code doesn't get tested.
Name one testing tool you use
BY: Zope.testing, also not zope.testing
BD: py.test, doctest
TC: py.test, doctest
MT: Abbot (for java swing)
KM: Buildbot
IB: py.test, nose, twill
NN: unittest, but I don't like it
IB: unittest, idea and eclipse, refactoring in java is nice
CW: nose
TB: nose, buildbot, paste.lint