Matt Harrison's Blog

Announcing Effective Pandas

2021-11-23T00:00:00-07:00

TLDR My new Pandas book, Effective Pandas, is released ebook here, physical book to follow soon.

You Wrote Another Pandas Book?

Yes, I wrote my third (or fourth, if you count Machine Learning Pocket Reference) Pandas book. In 2016, I released Learning the Pandas Library, and in 2020 I released the second edition of the Pandas Cookbook. Why would I do that?

I think my books fill a need. Often that need is to write the book I would have wanted when I was learning said technology. Learning the Pandas Library is almost six years old and one of the first books on the subject. While not outdated, I have taught and used Pandas extensively since then, and have developed strong opinions on the correct way to use it.

About two years ago, I had most of the book updated for a second edition. It had new information, cleaned up some areas I wasn't happy with, and added assignments. That book sat in Git and never saw the light of day. Early this year, I planned on releasing it. Instead of releasing it, I re-wrote it. And created over 100 images. (Writing software to generate and layout many of them.)

The result is an highly-opinionated book that teaches best practices for working with columns, manipulating tables, summarization through grouping, pivoting, and cross-tabulation, plotting, and focusing on writing clean code that your future self (and colleagues) will appreciate and understand.

I was asked a few questions in the process and thought I would answer them here.

What does Highly-Opinionated Mean?

I named the book Effective Pandas because a lot of material out there (on a certain blogging platform particularly) claims to teach Pandas. However, much of the material is not effective. It is confusing, bad advice, or often just plain wrong.

I am a corporate trainer and have taught Pandas to thousands via small group trainings, larger virtual workshops, and a course I teach for Stanford continuing education. I see a lot of my students confused or coming up with bad solutions because they are adopting the (poor) advice easily found on the internet.

My conference "tour" this year was mostly focused on some ideas that can make your Pandas code better. Many of these talks are kicking around the internet. Here is one. My book summarizes these thoughts and provides many real-world examples.

How is it Different than the Pandas 1.x Cookbook?

The original Pandas Cookbook was written by Ted Petrou. I read a lot of Pandas content, and I read this book. I thought it was a good book. I was approached to "author" the second edition. I added a few chapters, re-wrote much of the code (using some of the ideas found in my new book), edited some parts, and released the book. In the end, this is not "my book", it is "my take" on Ted's content. Ted has some really cool content that is worthwhile. It is a great book, however, it is not the Pandas book that I wanted.

Do You Have a Chapter on Extending Pandas?

Nope, I don't. I have never had that need, nor has a client of mine. If I had unlimited time, I would include more material.

I have heard that this is difficult, and there is not much material covering it. I can commiserate with this. I spent a lot of time on the time and date sections of the book. And while there is some material floating around the internet, best practices are hard to come by, which can be frustrating.

I also went down a rabbit hole of exploring all the combinations of grouping, pivoting, resampling, cross-tabulating with functions that return scalars, series, and dataframes. In the end, it was an interesting thought experiment. However, I don't think that exhaustive treatment was appropriate for the book (even though it is not available anywhere, except for some hand-written notes on my desk).

Do You Have a Chapter on Using Pandas with Django?

No. See above.

Why Self-publishing?

Self-publishing this book lines up with my goals of corporate training. I want to be able to easily include this with training material. I'm positive I could have multiple contracts for this book if I wanted to.

Isn't Self-publishing a Pain?

I've worked with two different publishers and self-published many books. There are pros and cons to each approach. In fact, I get asked this question so many times that I created a course, Effective Authoring, about book creation. It discusses these topics and interviews over a dozen technical authors to get their take on this and other questions. I'll answer it this way: it is a different pain than working with a publisher and you need to be able to manage some of the aspects that a publisher does for you.

What is Your Process for Writing a Book?

Here is a simple outline:

Seriously contemplate whether you want to use your time writing a book.
Outline the book.
Research (or use experience) to write code that demonstrates the outline.
Write text to explain the code.
Create imagery to reinforce concepts.
Flush out details.
Get feedback (my tech reviewers were great!).
Release.

Again, my authoring course goes into detail about book creation. I use Emacs. I write in a (sub/super)set of reStructuredText that allows me to create PDFs and ebooks from the same content. I leverage reviewers and automation (Grammarly) to clean up and clarify my content.

How Much Time Per Chapter?

This is one of those trick questions. It doesn't take into account that I have used tools for a long time, and thing that are difficult for beginners become second nature when you do it often. Some of the chapters have content I have taught so much that I can write it from scratch with little effort. Other chapters took multiple days to review, test, and summarize.

I could crank out the text for most chapters in about a day (the book has 35 chapters). I wasn't working on this full time, but would work between trainings and during downtime. There are a bunch of images that are time-consuming to create. (I like to batch those as it seems more efficient.)

One of the images from the book. Creating content like this takes a log time, especially if you make almost 100 of them.

Then you have to do edits. Tweak code, tweak examples, clarify. It takes a bit of time. There is a lot of bouncing around, which is necessary but the context switches slow down the process.

I Use Google to Search for Pandas Recipes, Will this Help? Is it Better than the Pandas Docs?

It is hard to answer the will it help question. Not all content pleases everyone. Putting your creations online can be nerve-wracking, and reading your Amazon review can make you want to go live off the grid and never show your face again. I've come to accept that you cannot write a book that satisfies everyone's needs.

Many of the "recipes" floating around are bad advice or wrong. If you read this book, it might help you determine if the advice you find is actually good.

Also, I spent a lot of time creating the index for this book (I feel this is one of the most important features in the physical book). The index is fifteen pages long. (The index for Learning the Pandas Library was five pages long as is the index for Pandas 1.x Cookbook.) Some people prefer to use analog indices to digital ones.

Regarding the Pandas docs question. There is some good content in the Pandas documentation. But the official documentation has different goals. It needs to be a reference for every function and method. I don't need to be a reference, in fact, I skip (on purpose) some of the functionality because I strongly believe that your life will be better if you didn't know it existed.

Also, the Pandas docs often demonstrates functionality with random data. For me, I understand the concepts better with real data. My book is full of interesting (to me at least) data, not just random numbers.

Finally, the Pandas docs usually show single method calls. It does not show end to end analysis, or what code in the wild might look like. Again different goals. I'm trying to help practitioners write better code, not document everything.

As I said earlier, if days were 50 hours long, perhaps I could include the whole kitchen sink in the book. However, my audience is not someone who wants to memorize or learn about every single option found in Pandas. My audience is someone who wants to write code to manipulate data but also wants to revisit their code in a month and clearly understand what it does.

Do you Still Like Pandas?

In addition to corporate training, I also consult around Python and Data Science. I use Pandas quite a bit. I was initially attracted to it because I had written a similar library back in 2006 for business intelligence reporting. Since then, Pandas (with all of its warts and inconsistencies) has taken the world by storm (for better or worse).

I consider knowing Pandas an essential skill for anyone working with structured data (in Python). It has its problems, and I don't shy away from them in this book. At this point, I think Pandas the API is more important than Pandas the library. What does that even mean? Well, there are more than a dozen libraries that implement the Pandas API. The API has become the defacto Python interface for data (somewhat like SQL). Many SQL people would tell you that not all SQL is the same, and it is possible to write clean SQL and poor SQL. I feel similar about Pandas.

Do I Submit Issues?

I have submitted issues to Pandas and did so for this book. I found a regression during the Cookbook, but that didn't happen during this book.

At the End, Was it Worthwhile to Write a Book?

The first step of writing a book is serious contemplation about the value of your time. My first book (what is now Illustrated Guide to Python 3) needed to get out of me. However, at that time, I was employed.

At this point, I think I'm a little more strategic. I run a training and consulting shop. I consider myself an educator and really enjoy corporate training. As much as I have tried to "crack the nut" for selling corporate training, I haven't found a repeatable sales process. Most of my clients hire me after reading a book or listening to a conference talk.

I realize that there are "training shops" that offer hundreds of different trainings. If I was buying training Python or Data Science, I would be wary of them. I wouldn't want to have a professional slide reader training my team. I would rather have an expert that can discuss the pros and cons and knows what the warts are because they have witnessed them first-hand.

My books are my sales and marketing team. It has been said to invest a good deal of time into marketing, and I believe that has been the case with this book.

If you are considering writing a book, you should think about the best and worst outcomes. Most of the reporting from book sales you hear about is probably from the top end, so don't be disappointed if you don't achieve similar results. Writing a book in and of itself is a great way to master the material in the book. It can also be a nice introduction, "I wrote the book on that."

Checkout Effective Pandas here, physical book to follow soon.

What is the best free course?

2020-09-24T00:00:00-06:00

TLDR You need to have skin in the game and invest in yourself. If my courses are too expensive for your part of the world let me know what you can invest in yourself.

Blockage

Last week something new happened to me. I was blocked on Twitter. It has probably happened before, but I never noticed it. The blocking happened when I asked an executive from a VC-backed company that has raised over $100 million why they were asking for free labor.

I have nothing against helping people. What bothered me was that they asked people to donate 8 hours a month to a commercial venture. Maybe my response was a little snarky, but it resulted in a blockage, and I never even got my answer... why do they need free labor?

Why are you are asking for free labor? 🤔
— matt harrison (@__mharrison__) September 16, 2020

Free free free

Come to think of it; I get asked a similar question often: "What is the best FREE course on Python (or Machine Learning ...)?" I also get a variation on that "Will you give me your [Fill in the blank] course for free?"

I don't think there is anything wrong with looking for a good deal, but I generally don't do free. First of all, I feed my family selling snake oil. (Ok, not that snake oil, but I teach people to be productive with Python and Data Science and generally get good results based on feedback and repeat clients.) Would you ask a car dealer to give you a car for free? Also, asking me (a biased producer of content) to recommend a competitor seems a little tacky...

"But Matt, you offer virtual products, it doesn't cost you anything! It is not a car!"

Don't be cheap

Don't be cheap. A close family member of mine is cheap to the extreme. They buy the cheapest food and only splurge on the certain day of the week when it is on sale (splurging is being generous here). The modus operandi is doing whatever requires the least effort. They consistently refuse to fix anything that is broken, or they try and find a "handy" neighbor who is too nice to say "no" and convince them to fix their problems (for free!). This has resulted in a horrible quality of life and has impacted close friends and family. These problems are not due to a lack of money, but a lack of placing value in anything (and untreated mental conditions).

"You wouldn't even know a diamond if you held one in your hand" - Steely Dan

This has taught me that you should be willing to invest. Often the cheapest "solution" ends up being more expensive in the long run.

Put in the work

You value something that you have invested in. A couple of years ago, an acquaintance lamented their long-hours at a poor-paying job. They were convinced they wanted to be a developer. I told them they would probably double or triple their current salary when getting their first job. I could see their mouth watering. They asked me how to proceed. My advice in rank order:

Get a degree
Go to a bootcamp

No, they wanted to be cheap. They didn't have the time. They didn't have the money. Ok, I gave them a copy of my book and said, let me know when you've read it and I'll give you more advice. They never ended up reading my book. But every time we ran into each other, they talked about how they wanted to be a developer... At some point, some 3 years after our initial conversation, it was the same thing. They wanted to be a developer, but they never invested.

"Some things will never change" - Steely Dan

At that point, they could have had a degree. Had they gone the bootcamp route (and gotten a job, which I'm pretty sure I could have helped them with), they could have had two years of salary plus raises. The opportunity cost was approaching $200,000.

"Enough with the anecdotes. I want free.", you say. Ok, as I said, I am not against helping out. If you want to learn Python for free, the documentation at python.org is good.

"No I want a free course". Go to youtube and search. There are many free courses. My experience consulting for a company providing backend software for MOOC's tells me that 95+% of free course attendees drop out. They have no skin in the game. Like my friend, real life, netflix, or something else distracts them from investing in themselves.

Realize that most "free" things are trying to sell you something. There are exceptions, but this is a general rule.

My Courses

"When black friday comes" - Steely Dan

"I want your course for free." I have discounts on my mailing list and on Black Friday I also discount my wares. I generally don't do free. If you are asking for a free course, I'll probably respond with, "what can you afford?" My experience is that if someone is not willing to invest, they are not a good student. Admittedly my course may seem expensive outside the US. Hence, let me know what you can afford. I can create coupons.

"I really want your course for free." Note that I do offer review copies of my course to reviewers. If you think you qualify as a reviewer you will need to provide evidence of that. Blogs, Amazon, etc. (You probably aren't if you are asking for a free course.)

Also, I do offer my courses as prizes to relevant meetups and user groups. You would not get my course, you would need to be an organizer of said group (and provide proof), and I would provide a coupon in return for a quick "thanks to our sponsor" blurb at your meetup.

An Inverse Incentive

I've thought about offering these "free" folks an incentive. Upon completion of the course I refund them. However, I think I would feel bad because I'm pretty sure most of them would not finish the course. Thoughts?

Best of luck with your learning process. Hit me up on twitter if you have any advice for "free courses".

My New Course on TalkPython

2018-04-26T07:32:00-06:00

Following the release of Illustrated Guide to Python 3, my friend Michael Kennedy approached me about creating a course for Talk Python. (In his concerted effort to wean people off of legacy Python) He wanted a course with all the new features of Python 3. We planned it out, and I initially thought it would be a short course, on the order of an hour. It turned out there were a few more features in Python 3 than that.

The final course Python 3, An Illustrated Tour, came out with just over five hours of content. Much of this content has very little prior material covering it (books, videos, or even training courses). It includes the usual suspects: the print function, Unicode, and long unification. It dives in pretty deep into asyncio, annotations, annotation tooling, and pipenv. I even went and purchased a Windows machine to ensure that the environment section worked on Windows.

(As someone who was exclusive to Linux from 1999-2014, this may seem abrupt. Or maybe not. I've dealt with Windows quite a bit in my corporate training (and assorted family tech services). I purchased a Surface book and recorded most of the videos on it using to pen to annotate and add drawings. Happy to report that Python runs quite happily on said machine.)

There have been some questions about how this course is different from my book Illustrated Guide. I think the course description does a pretty good job, but I will elaborate a bit here. The book is meant to introduce Python. As Python 3 is the modern version of Python, that is what it covers. The book will provide an excellent foundation for someone who is technical and wishes to learn Python. This course has a different audience. It targets someone who has been using Python (either legacy Python or Python 3) but is programming it in the common subset of those versions. Python 3 has some cool new features that are not commonly used (or known).

The course covers the new unpacking syntax (both generalized unpacked and extended unpacking), dives into keyword-only arguments, teaches how to use pipenv (on Windows as well as Unix platforms), covers asyncio (including generators, context managers, and iterators), and presents annotations and the tools used to take advantage of annotations. Also, it offers some of the new packages in the standard library, enum and pathlib.

In addition, there are hands-on labs to help solidify your newfound knowledge. These labs are self-correcting, enabling you to try things out until they successfully run. Each of these labs has a walk-through as well, so you can see how your solution compares to mine.

As a bonus, if you buy the course in the next week, you will get the book as well! Head on over to the course page and check out the contents!

"Kickstarter for Python and Data Science Screencasts"

2015-06-02T23:30:00-06:00

Down the Training Rabbit Hole

In the past few months I have been doing a bit of corporate training for MetaSnake. I have noticed a few things:

Programmers that have used Python in a work setting miss details that I consider "beginning" knowledge
Programmers that are new to Python keep coming back to things that I ignore or don't think about (type checking, attribute protection, compiler warnings, white space)
Most content explains how to do something, but neglects the errors. The only recourse is Stack Overflow. Newbies get stuck (and frustrated) on errors
There is often more than one way to do something
What is Pythonic and "idiomatic" in (C)Python is often not in the Data Science stack
Data Scientists using Python often use poor programming practices

Introducing Pycast

To that end I'm Kickstarting a new site called Pycast. Pycast will provide subscriptions to two channels, a Python channel, and a Data Science channel. users can subscribe to one or both.

My aim is to publish content in both channels on a weekly basis. I want it to be at a reasonable price. One month will cost about as much as a lunch (US). The content will cover one topic. It will generally be under 10 minutes, be very code heavy, and show hints as well as hangups.

The Python channel will cover topics like:

Beginning Python - using IDLE, variables, strings, functions, classes, etc.
Intermediate Python - comprehensions, closures, decorators, generators, magic methods.
Testing - Unittest, doctest, coverage, CI, linting, generative testing, and more.
Environment - Using virtualenv and pip. Reproducing installs, and automation for testing and clean installs.
Packaging and Layout - How to layout a package, get it up on PyPi, etc.
Standard library - Looking at common modules, and diving into the source.
3rd Party - Review popular libraries, usage, coding styles, etc.
2 vs 3 - Continue stoking the fire here. Writing code that supports both.

The other channel I want to start off with is a Data Science channel. I've been doing "data" in Python for a while now. There is a lot of demand for Python in this arena. While data science is a somewhat ambiguous term, my content will cover:

Collecting data - Scraping, crawling, and automation.
Inspecting data - Basic summary statistics and visualizations.
Munging data - The fun janitorial work that takes up too much of my time.
Analysis and Machine Learning - Classification, regression, clustering, etc.
Visualization - Both presenting the data, and analyzing the analysis.

Why not one channel?

Though there is some overlap in the content (hopefully Data Scientists are idiomatic Python programmers), I think the channels will appeal to different groups. I've certainly seen that in my corporate training gigs. Though I might have some videos that belong to both channels (SQL is an example).

Why Kickstarter?

First of all, I want to try running a Kickstarter just to have experience with the platform. It is a nice way to launch a product. I also want to attract a group of users who are interested in getting early access and providing feedback. With the aim of having a better product when I launch to the world.

If this is of interest to you, I encourage you to join for a month, or a year. (If you or your company need training, I've also put in some packages that basically give you a half day or day of training with a year-long subscription thrown in).

As of this blog post, I'm at 10% funding. Please spread the word.

"Using Decorators for Testing Screencast"

2014-06-18T07:32:00-06:00

I made a screencast exploring one of Raymond Hettinger's tweets a while back. His 140 character exposition suggested that a student of his wanted to use decorators to unittest functions.

#python idea (from a student): @test(inputs=(5,2), output=25) @test(inputs=(2,5), output=32) def mypow(base, exp): return base ** exp
— raymondh (@raymondh) April 25, 2014

In the screencast I write a decorator that does that from scratch, dicuss "module load" time, "run" time, the timeit module and doctest. Enjoy.

"Revisiting the ErgoDox"

2014-05-13T10:41:00-06:00

I purchased and built an ErgoDox keyboard over a year ago. I thought it would be good to revisit that since I get asked questions about the keyboard every now and then.

What is an ErgoDox?

The ErgoDox is the Voltron of keyboards-a combination of open source hardware and software, that when combined with a little elbow grease from the user becomes a powerful keyboard. That description is not too far from the truth. There currently is no way to buy a complete ergodox keyboard, one must purchase (and assemble) the parts piecemeal or buy a kit from a company such as MassDrop.

What is it?

To be more precise it is keyboard that has the following features:

Split keyboard halves - There is a piece for the left hand that is completely independent of the right hand piece. This allows a good deal of flexibility in fitting the keyboard to an individual. I can position the pieces close together to emulate more traditional keyboard configurations, "tent" them for a more natural feel, or place the wider apart. My current favorite is to have them about shoulder width apart. To get an idea of what that would feel like, place your hands on the outside edge of a 13-14 inch laptop. I really like that positioning currently, though was tenting with about six inches between previously.

Hacking at the hacker space pic.twitter.com/tf7n3BOt1A
— matt harrison (@__mharrison__) May 7, 2014

There is even an open source project for tenting it.

Matrix layout - The (finger) keys are oriented in columnar fashion, unlike normal "staggered" arrangements that are oriented in a slightly up and to the left manner. Other than staggered layout being the norm, I don't see how anyone could argue that it is ergonomic in any way. There is some hardcoded vertical offset of the columns but it seems to work for me.
Thumb clusters - Both pieces have a cluster of keys for thumb use. Most people only type with one of their thumbs, which is a shame. If they do use both thumbs they are likely using both for the same key. I don't know that pianists do this? On my keyboard I'm using Control and Meta on the left thumb (emacs user here), and Enter and Space on the right.
Mechanical Switches - Mechanical switches such as the Cherry switches are rated at 50 million presses before failure. They also have switches with different mechanical properties-activation forces, tactility, and sound. Having broken cheap dome switches, it wouldn't make sense to invest the time and cost to build a keyboard only to have the switches fail in a year.
Open Hardware - On that note, the hardware design is completely open, I could replace a switch, or teensy microcontroller if I needed to. This is empowering in an era where hardware is disposable.
Open Software - The firmware is also customize-able, in fact requires it. Indeed, the lack of labels on the keys (at least in the MassDrop kit) and having keys that exist in positions that do not occur on traditional keyboard layouts encourage some thought about what an ergodox layout should look like a particular user. There are now two main firmwares for driving the keyboard. The traditional fw, and the tmk fw which has been ported to support the ergodox (and provides features such as macros and USB mouse emulation). Both fw's have the notion of "layers", where a new layer be it qwerty, colmak, dvorak, custum gaming layout, or programming layout can be turned on with either a chorded keystroke or a toggle. MassDrop has even gone so far as to provide a web gui to customize and share layouts (though it is lacking at providing the complete power of the fw and is not open source).

Status

I originally bought a single keyboard to use at work. I found that I enjoyed it so much (or at least convinced myself that I did), that I purchased another keyboard for home when the second MassDrop buy appeared. I liked having a backup and at the time wasn't sure that there would be an easy way to buy one again. (In reality MassDrop has put them on sale every few months). Both have the smaller case that lacks the built-in wrist rest. My original purchase has "blue switches" (tactile with sound) as I was coming from a Unicomp Buckling Spring keyboard. The second has "clear switches" (tactile with less sound) so as to not annoy the wife so much with the clackity clack.

Construction was pretty straightforward for a soldering newbie like myself. It took about six hours to get the 100 or so parts attached to the first board. It then took a while to debug and fix a few of my connections. I thought the second would be done quicker and indeed I proceeded at such a fast pace that I put half of the diodes on the wrong side (the PCB is dual sided and I wasn't paying enough attention, or it was really late at night...).

Being a kit there were a few niceties that were lacking. I added rubber bumper feet to the bottom, purchased "nubbed" locator keys, and bought a mouse wrist rest for each side. I also wish that the TRSS cables were a bit longer to allow for more placement experimentation.

From a few days in my layout has been the Norman layout. I grew up typing on computers with qwerty, and felt comfortable with it. Given that I was going ergonomic, I thought that it made sense to go to a more ergonomic key layout. Because the fw is in the keyboard, I can plug it in and start typing on any computer with a USB port. No need to install software. I always thought I would add a qwerty layer, but have never got around to it.

In addition I have a USB mouse emulation and arrow layer and a programming symbols layer. These last two are newer as I recently migrated to the tmk fw. I use them a bit, but have not been fervent about using them. I normally place a mouse between the halves.

If there were more time in the day I would optimize it more. I have loaded up some of my work into some Pandas dataframes to view how my key frequency compares to standard frequencies. Coming up with a layout is always a compromise and could involve endless tweaking.

Norman was painful for the first week. The second week was just slow, and after that I seem to have adjusted ok. I don't know that my typing speed has improved any over qwerty, but that wasn't my goal. I did do some online Nitro Racer typing races for a bit, but after a month or two left it at that. My qwerty is ok still. It has suffered, but I can switch back pretty painlessly. My original intent was to treat the new physical layout as a new "instrument", say guitar vs bass, and I think it has worked.

The blank keys help during the learning phase in that they discourage looking at the keyboard. They also provide some obfuscation to onlookers who might want to see your password, or prevent your kid/wife/anyone else from using your computer (my tiling wm already did this though). I find that the locator (ie "nubbed") keys were required for my hands to locate themselves correctly.

I have also created a case to carry it around in. It is fashioned from a small plastic (pistol?) case with foam cut to the shape of the halves. My biggest concern is not transportation of the keyboard per se, but the insertion and removal of the connector cables (TRSS and micro USB) every time I move it. Some in the geekhack forums have complained about faulty TRSS connections.

@kejadlen See pic.twitter.com/e6iGNbXYjY
— matt harrison (@__mharrison__) May 8, 2014

Miscellany

I know of at least one co-worker who has an unassembled kit sitting around for almost a year now. Another buddy claimed that column orientation threw him off (but then he went off and put Norman on a staggered layout!). Keyboards can make you do wierd things.

Is the ergodox the perfect keyboard? No. It is hard to even acquire. You need to wait for a MassDrop buy, or purchase parts piecemeal. And then there is the cost. It isn't cheap-$250 for parts plus time to build, customize, and program. Even though it is pretty ergonomic, Kinesis and Maltron users with "well" keyboards tend to really like the wells. And the thumb cluster location is also hardcoded which throws some people (indeed I only frequently use two thumb keys out of six on each hand).

There is also an onramp time for both physical position of keys and layout of the symbols/letters that are in a non-standard location.

Also inspiring is Phil Hagelberg's keyboard that he constructed after making an ergodox. One would hope the combination of smarts and 3D printer's would allow for truly custom keyboards in the near future. Do I regret my purchase? Nope, and I don't know that there is another keyboard out today that I would consider over it.

"Subway Map to Python"

2014-05-06T10:23:00-06:00

Starting something new can be overwhelming and confusing. Where does one start? Where does one desire to go? Because Python (the language, standard library, ecosystem, community, and more) has a such a wide breadth of functionality, it can be even more daunting.

I have long been impressed by the abstract artistic quality that subway maps present while still illustrating a path to a destination. In an attempt to provide a newbie to Python with a similar experience I attempted a "Map for Python".

From an artistic point of view, I think it looks more like a melted and refrozen snowflake. Like a map, there is some abstraction, and it is in no way complete or up to date. There are many paths through the Python language, ecosystem that surrounds it, and communities.

There are layers that could go on top linking projects, diversions along paths, and more. What would you put in a "Path to Python"? I have probably missed your favorite project.

For those interested in getting on the path, this month I will be selling a course-90% of Python, that will quickly bring an individual (or a team) to the Beginning point and allow you to jump off into the vast world of Python.

"Introducing 90% of Python Course"

2014-04-25T09:48:00-06:00

In addition to some consulting work and catching the tail end of spring skiing, I've been working hard. Early next month I will be selling a course-90% of Python.

A course supplemented with both videos and exercises == HUGE win. I'm a visual learner... the videos help explain further and drive home the point/purpose of the exercises. - B Morrison, Satisfied Customer

This is a multimedia course aimed at professionals who want to learn Python quickly. It includes:

2+ hours of video training
Best-selling book Treading on Python
Self-correcting labs to allow practicing Python
Video explanations of the labs
A cheatsheet with all the basics of Python

Why 90%? I want 100%!

Python is very broad. The intent of this course is to present the language, nuance, and best practices around Python such that you can go out and read and write Python. Then there is a huge standard library, third party packages, fancy constructs such as metaclasses, decorators, and generators, C integration (Java/C# too), web development, scientific applications, thick clients, automation, and more ... This course will provide a foundation for the rest. Covering the rest could take quite a bit of time, but with the basics you can be productive quickly.

Team (2-Pizza) Edition Offered

In addition to individual licenses, there will be a 2-Pizza license (for teams the size that Jeff Bezos recommends). If you want your team to be on the same page, let your boss know to pick up a team license.

I will offer a promotion during the launch only, sign up on the landing page to receive the discount code.

"Cyber Monday Sale"

2013-12-02T06:53:00-07:00

The kind authors of Two Scoops of Django and I have teamed up for a Cyber Monday Sale. Check it out if you are interested in mastering Python and Django!

"How I Spent Three Years Writing a Book"

2013-08-13T08:15:00-06:00

As blogger James Altucher would say, I finally have a stack of business cards. Physical business cards that I can hold in my hand! OK, they aren’t really cards, they are books. Books the I have written over the past 3+ years. I’ll try to always have a set of books nearby to use as a business card.

Why did I write a book? My original plan was to convert some popular tutorials that I had given at PyCon into book form, when I was given the chance to present one at OSCON a bit later. Rather than just rehash the slides, {"I figured I’d write down the book that I would have wanted when I was learning Python. I didn’t just want the syntax, but the Pythonic constructs, the do’s & don’ts, the beginner mistakes, and the community best practices that are gleaned over time."} The result was “Treading on Python Volume 1” and that book took a lot longer to write than I planned (had to learn about ebooks, their creation, and formatting them so they actually look nice for code on all of the readers). And I had to add in all of the 3.x’isms that beginners might want. The ebook version was released in late 2011.

Ebooks, ebooks, ebooks

In the process of releasing that book, I had bikeshedded my own tool, rst2epub2, for creating ebooks from a lightweight markup language - reStructuredText. I have even taught two courses to Elementary aged students about the process of ebook creation and self publishing. Teaching them HTML and CSS (the main ingredients for ebooks) as well as publishing a physical book with a collection of their stories.

Following that, I began working on the second volume, this time covering intermediate topics. Information in book form that I wish existed years ago. These are some of {"the cool features of Python that you can get by without, but you really do not want to"}. You should learn them and take advantage of them. This time covering:

Closures & Decorators
Iterators & Generators
Functional Programming and List Comprehensions

Each of these topics I ended up releasing as a separate ebook, but also combined them all into Volume 2, which was released earlier this year.

The Desire for a Dead Tree

If you browse my blog, you’ll see that I had intended on getting physical books out awhile ago. I had a proof from Createspace made in May of 2012 and even paid out $200 to add support to Sphinx for better LaTeX generation. In the end the bike shed that was Sphinx, wasn’t painted my shade of pink. So physical books had to wait a bit longer. I spent a couple weeks actually diving into LaTeX and learning it. Then I created my own tool for creating physical books (with proper front matter and back matter) from reStructuredText (rst2nitrile).

My PDFs are not perfect, but I think they are much better than many self pubbed attempts. I’ve learned enough about books, typography, layout, design, etc over the past three years, that I doubt any book I do will be perfect (I don’t have the time to make them perfect, with a full time job and a family). But they are good. They look good, they follow best practices, they have indexes, and everything is done via the command line. My tools have basically abstracted book creation so that I (almost) have single master source that converts to pdf, epub, and kindle. If any of that interests you, all of my tools are open source, and I will soon be posting more details about them.

To celebrate the pile of books that I can now hold in my hand, I’m putting them on deep discount on Amazon for the next couple of days. (Feel free to tweet that out, facebook it, pinterest it, instagram it, vine it, or whatever the social sharing mean dejour is). If you enjoy the books or find them useful, the best thank you I can receive is probably a kind review on Amazon. If you find issues or areas where you feel they are lacking, drop me an email. My toolset is agile. I can push updates easily.

Special Offer 1

If you run a User Group, please email me with a description of the group and I will get you some copies of my books to give away. (Doesn’t have to be a Python group).

Special Offer 2

{"Amazon has Auto-Rip for CDs, but nothing similar for books."} I’m willing to provide that as an experiment. If you purchase a copy and tweet a picture of you holding the physical book, I will reach out to you to get you a epub/mobi/pdf copy as well.

Note, if you have an allergic reaction to Amazon, but would still like a copy of the book, I have “review” copies available. Also, I will be making epubs and PDFs available for purchase soon.

In summary, writing a book is a lot of work. Doing almost everything by yourself, including creating the tools to create the books is more work. Doing that while holding down a job, and family life takes time. This isn’t a sprint, it’s an ultramarathon, and I’m bushwacking the course. Thanks so much to my wonderful editors, JJ, Amjith, and Luke.

"Life Is Busy"

2013-07-11T06:25:00-06:00

Here is an update on what I have been up to in the last while.

Python Related

I have been approached to do some corporate Python training. I would love to do it but real life has just gotten in the way. I really enjoy helping others learn (hence my involvement teaching Tutorials at PyCon, being a co-chair of Utah Python, writing books, etc), and would love to incorporate this into a bigger portion of my life, but with a family, a full-time job, and other hobbies it has been hard to do much more.

On that note I'm giving a tutorial on Python/Pandas/Matplotlib at the Strata Conference in NYC this fall. I had so much fun at it and PyData last year (barely squeaking out before the big storm), that I wanted to go back. Strata is a "big data" conference, and there is a focus on Hadoop and it's ecosystem, but there are also tracks around machine learning, visualization, and "in-core" data. I'm not sure if PyData is happening in NYC the same time, but it should be a good time. Since I've been doing Pandas and Matplotlib pretty heavily at work, I think it will be a really good tutorial.

I have also been putting in spare time into finishing up Treading On Python Volume 2 and Guide to: Functional Python & Comprehension Constructs. Volume 2 is composed of my Guides to:

Decorators
Generators
Functional & Comprehensions Constructs

They have done ok so far. During the release it was #1 on Amazon for a while, this screenie was after it was bested by Javascript: The Good Parts. Not too shabby!

#2 in Software Development on Amazon! Thank you readers! pic.twitter.com/3f4D3EZQbS
— matt harrison (@__mharrison__) May 2, 2013

I also released an ebook on formatting ebooks. It got a mention from Pydanny's "Two Scoops of Django" as it proved useful to him during the creation of his ebook.

View from alternate reality where code is properly formatted in mobi7, kf8, epub (ibooks,nook) #eprdctn pic.twitter.com/FRSEIMUX
— matt harrison (@__mharrison__) January 18, 2013

My Kindle books are available here. I also have a few review copies available for folks who are interested.

In addition, I have been learning the ins and outs of LaTeX. Finally (I had a proof a year ago of Treading that I was not satisfied with) I have held a physical printing of my book that I am satisfied with. I should have dead tree books for all of my python books available soon. If you run a user group and would like a copy to give away at your meetings get in touch with me.

One of the reasons it took a while to get the physical book out is that the available PDF generation tools for rst were not painted the color I wanted them. Sphinx is great for a website, but not for physical books nor ebooks. Docutils does not do ebooks and is lacking features such as indices. So I have written tools to scratch my itch that allow basically single source rst to be transformed into ebooks and physical books. They are available at github/mattharrison. I apologize for the alpha/beta state, they satisfy my needs, but should get cleaned up a bit.

I have also been messing around with my raspberry pi's. I have been using one with XBMC. Being that I do not watch TV that much it really has not gotten much use. The Python channel is interesting and PBS on demand is cool. The interface is a little slow, but I think that is mostly due to my slow internet and maybe a lack of caching. Lots of 10 second delays here and there. It also does not quite do airplay without the neato realplayer buffering effect that we dreaded 15 years ago...

My other pi is in my garage running my sprinkler system. I ordered an OpenSprinkler add-on and have been pretty satisfied with the results. My wifi dongle appeared to lose it's connection over time and things have been better with ip over power. Though it still has become unresponsive every now and then. Right now it is on a nice little schedule (open sourced by a kind individual), in my free time I'd like to add the ability to lookup current temp and precipitation and adjust the schedule dynamically.

Other Stuff

In addition to my marathon slog of book writing, I have run my first marathon. To be correct I actually ran the Squaw Peak 50, a 52-miler (got lost along the way and ran two extra miles) with a buddy in June. After pacing my buddy for a portion of the Wasatch 100 last year, he convinced me that I needed to try my hand at endurance sports (not sure why I agreed since my pacing of him was one of my longest runs up to that point).

Up high on the race overlooking Utah valley pic.twitter.com/gU1zCt3cBf
— matt harrison (@__mharrison__) June 2, 2013

We finished in 13 hours and had a good time. Both lost a toenail. So I have been training a bit for that. One part of the training was running the Zion Ragnar with the nice folks at Red Brain Labs.

#zionragnar is done pic.twitter.com/tmslnlqsbU
— matt harrison (@__mharrison__) April 28, 2013

I tried to get in a bit of skiing with the kids in the spring and went backcountry with a buddy. I'm now really excited for it to snow again and am kicking myself for not doing backcountry earlier.

In addition I have been playing a bit of ultimate and trying to ride my mountain bike.

Keyboard

I posted a bit back about the ergodox. I liked it so much that I ordered another. I settled on the Norman layout and am up to around 45 wpm on it.

Up to 25 wpm #ergodox pic.twitter.com/82OK1pWEld
— matt harrison (@__mharrison__) April 18, 2013

Family

I've also been busy with the family. We have had some good trips to Goblin Valley, two to Wyoming, camping up in the canyons, etc. Just trying to have fun and enjoy the summer.

Life is busy but it is good.

Goblin Valley... pic.twitter.com/jRihe7G8TU
— matt harrison (@__mharrison__) May 25, 2013

"Oh (Ergonomic) Keyboard, Where Art Thou?"

2013-04-02T07:26:00-06:00

TLDR I have found the perfect keyboard (for me) that has the following features:

Ergonomic
Mechanical Switches
Split
Matrix
Thumb keys
Programmable
Open Source (Bonus)

(My new keyboard on my standing desk stand (raised by Logitech box to correct ergonomic height). Old unicomp in background)

I’ve become a keyboard snob

After building my first computer some moons ago, I soon became interested in ergonomically designed keyboards. Perhaps the fact that they were different from normal keyboards was what drew me to them initially. These differences led me to ponder-why would anyone want a keyboard that is shaped differently than a normal keyboard? That led to me learning about RSI and ergonomics. I started off purchasing a cheaper logictech keyboard that I found at Software and Stuff for use at work. Surely if I was going to make my living typing code I should invest in my equipment to protect myself. Besides, how would I throw a frisbee and play ultimate if I had carpal tunnels?

Time passed and I picked up a used MS ergonomic keyboard. It seemed pretty good other than the arrow keys were arranged in a cross pattern. (Why change that yet leave everything else in the same position? It boggles the mind). I seemed satisfied and life went on.

Eventually I picked up an MS 4000. It was available and the local office store and felt decent enough when trying it out in the store. It worked fine (or seemed to). Until I started looking into keyboards more. I learned about switches. Each key has its own switch that activates it. The best switches were mechanical buckling key switches. They had a distinct tactile “click” when the press of the finger activated the key. This was supposed to provide for more ergonomic typing because your finger didn’t try to crush the key, but rather could feel when it was activated. They also made a cool sound. And if you typed fast enough it was like having a little machine gun. (The sound also corresponded to the mechanical activation, so you could base your finger movements on that as well). Another great feature about BS keyboards was that they seemed to last forever. As I looked into them many people had IBM Model M keyboards from the 80’s that were still going strong. I had already destroyed the switches on one keyboard and had just figured that keyboards were disposable. (Which was a sad thought because it seemed like every “ergonomic” keyboard at the time had limited runs, so you never knew what your next keyboard would be).

Hmmm, but there weren’t any buckling spring keyboards in ergonomic designs. Wrong! IBM actually produced an ergonomic version, the M15, for a short time in the 90’s. The only problem was they didn’t sell well and they stopped producing them. Of course since they were BS switches they should still be around. Ebay to the rescue. Sadly it appeared that every other ergo keyboard geek had the same idea as me. The price for these rare keyboards if they ever do reach Ebay is pushing $1500! (There just happens to be one on ebay right now with a buy-it-now price of $1249, that’s a steal!) I love my fingers and wrists but apparently not that much. But BS springs intrigued me. Plus I was using ThinkPads as my main workhouse and came to appreciate the track point in the middle of the keyboard. I could mouse around without taking my fingers off of the home row.

It just so happened that the company that bought IBM’s BS equipment and processes, Unicomp, also continued to make a BS keyboard with a trackpoint. Another key event of that time was that a key on my MS 4000 also went out. And after reading about BS switches I became very sensitive to typing on my “ergonomic” keyboard. (Finding out about BS switches was like reading “The Elements of Typographic Style”, once you know about kerning and ligatures you look for them everywhere. It’s hard to unlearn that factoid). I became very apparent that hitting the wider keys on the left edge with my pinky was a great pain. I believe this was because the keys are so wide that the keycaps are rubbing against the mechanism during depression causing binding and lockup. Due to that I would press them even harder, which is bad, and even worse for pinky fingers.

I figured if the current ergo designs have crappy keys, maybe BS switches are good enough and picked up a Unicomp Endura Pro. I was pretty happy with that keyboard. (My only gripe is that the buttons for the trackpoint aren’t BS and so they become worse over time or even stop working). But I loved the switch mechanism. It was probably half in my mind (Stockholm Syndrome) and half due to the crappiness of the switches on the MS 4000. If I ever had a keyboard that was in an ergonomic shape it would have to have high quality switches.

In the subsequent years I read about Kinesis, Maltron, and eventually Truly Ergonomic brands of keyboards. These companies were the only companies offering “ergonomic” designs in keyboards with mechanical switches. (None had BS switches, but they used Cherry brand switches, which many like because Cherry’s don’t just come in one style like BS switches-loud and tactile. Cherry’s have different switches that fill in a matrix with different values for activation force, tactility (or lack thereof), and loud (or not). Cherry Blue’s are supposed to be the equivalent of BS switches (or similar enough). The mechanical mechanism on these switches should be smooth and last for years, instead of months.

An interesting feature that I loved about these nicer ergonomic keyboards was that they used a “matrix” layout. If you look at any modern keyboard you will note that it slants up and to the left. If you think about that, it doesn’t make sense that both hands go up and to the left. Shouldn’t one hand do that and the other hand go in the opposite direction? And I haven’t even gotten to actual layout of the keys. Both of these “features” are hold-overs from days gone past of typewriters, where layout was designed to limit typing speed as to not jam the mechanism. In a modern age, where we have ergonomics yet also want to type fast shouldn’t we drop the legacy features of days past completely? Apparently not. Most keyboards still adopt the slanted layout. But the ergonomic keyboards I mentioned do not. Conforming to century old norms for keyboard layout has led to a lack of innovation in keyboards.

But all of these ergonomic keyboards had a glaring problem. They were “ergonomic” but hard coded the ergonomics! Similar to using a “magic number” in programming, they had physical plastic cases that forced (or assumed) the typist to conform to their ideal of button positioning and distance between hands. That seemed restrictive and silly. There were actually split keyboards that existed. Each hand was separate, but they all had crappy rubber dome switches and/or non-matrix layout!

What to do?

I ended up surfing across a forum for keyboard enthusiasts, http://geekhack.org/. One day in the ergonomic keyboard subsection there was a post. This was in October 2011. The post had renderings of a cool looking programmable, split, matrix style keyboard that would cost around $400. Pricy, but it had most of the features I wanted. Fast forward a year and a half later. After hacking/compromise of geekhack, (continued discussion continued on http://deskthority.com/), lack of community members interested in organizing a group buy, and http://massdrop.com/ stepping in to organize a group buy, I finally had a kit to build a keyboard. Fast forward a long night after than and I had my keyboard. It cost $240 and took about six hours to assemble (having never really soldered before myself).

(Device in action at my standing desk (plastic still has wraps on it))

Features include:

Mechanical switches I purchased Cherry Blue switches which are both tactile and non-quiet to emulate my buckling spring switches (my co-workers have yet to complain about my BS switches). They are rated at 50 million presses, which should outlast me. Having broken a couple rubber dome switches this was important. Also I appreciate that they are smooth and consistent, which has not been my experience with rubber dome switches. Once you go mechanical rubber dome keyboards feel chintzy. (The Lenovo and Apple laptop keyboards are pretty decent though).

Ergonomic A tenet of most ergo designs is that your hands are allowed to come to a natural position, rather than forced to fit into the confines of a keyboard based on a century old typing device. Check.

Columnar Layout Rather than a normal staggered layout like most keyboards, the ErgoDox has a columnar layout. Your fingers on both hands don’t naturally point up and to the left, and your keyboard shouldn’t either. (This might take some getting used to. Some columnar layouts put the number 2 above “Q” rather than one because your muscle memory seems to hit that better. I’ve just tweaked the configuration of my layout to try that scheme). Here's my current configuration.

Split Halves This was a biggie to me. Again, the Maltron, Kinesis, and TrulyErgonomic all look like they are decent keyboards, yet they hard-code the distance between the hands. With split halves I have 3 degrees of freedom for each half and can create infinite combinations. As the standing desk vs sitting all day debate is raging, I think there is a middle ground. Common sense seems to indicate that variety is probably better than either end of this dichotomy. You should probaly stand, walk, and sit during the day. Wouldn’t it be nice if you could position your hands close together for a time, and then spread them out further apart? Most keyboards don’t allow that. I’ll probably get some sculpy (maybe 3d printing) and experiment with stands for tenting and other angles of the keyboard. (The similar to the stand accessory for the Kinesis Freestyle2).

Thumb Keys Humans have opposable thumbs. Yet most people only use one thumb to type on one key. That seems to be a huge waste. Many of the more modern keyboards (Maltron, Kinesis, and TrulyErgonomic) have keys for the thumbs. This is probably the root of the cause for a steeper learning curve for these keyboards. Because many of the keys are moved under the thumb (and away from the fingers that normally depress them) it can make typing a chore until muscle memory is built. But I believe there should be big dividends after that time.

Programmable The folks at Massdrop created a nice little web application to create layouts for the keyboard. One cool feature is that these layouts can be “stacked”. You could create a layout for qwerty, dvorak, and colemak. You can let the keys under the thumbs be optimized to your desires. I’m still messing around with key configuration as I’m getting used to the keyboard.

(It took me about 5 hours to put this thing together. I'd appreciate a LinkedIn recommendation for my soldering skillz)

Open Source This applies to software and hardware. The firmware is on GitHub. The designs are in the open. Even though it took a commercial company to step up and organize the group buy, one could produce their own keyboard using the existing plans. Because I physically built this keyboard, I feel like I have some semblance of understanding of what is going on at the hardware level and could “fix” a problem with a switch or short should it arise (say a drink was spilled into the keyboard). I’m not sure that is the case with most keyboards out there today.

Yes, it is the most expensive keyboard I’ve ever purchased. But it is cheaper than both Maltron and Kinesis and costs about the same as the Truly Ergonomic. But I feel it is a better keyboard than all of those. I also flip that cost issue around and say it is an investment in tooling that should pay dividends. Also if the ergonomic benefits hold out, it is an investment in my body. Surely my wrists and hands are worth $240, which is about half of the cost of the original estimate.

Future Plans

Right now I’ve only used the keyboard for one full day at work. I’m tracking the commands I do in emacs. After a couple days of recording the commands I’ll examine what I’m doing and see if it makes sense to create a layer with emacs specific chords. I’m more cognizant of my finger movements, both when they migrate off the home row and with my thumbs. I’m trying to figure out what optimizations I could make to my layout. Since all the keys were the same, I threw on some bandaids the indicate hand position (yep need to order two more keycaps).

I’ll be honest. It is harder to type on right now.

One thought I’m having is geeking all out. Considering that many musicians can play a guitar and a bass guitar differently, I’m wondering if I should treat this keyboard as a new “instrument”. Rather than slightly tweaking the normal “QWERTY” layout, perhaps I should try MALT or Colemak. Considering that when I’m using the thumb keys it does feel like a different instrument perhaps it would be easier to make the mental switch between normal keyboards and my geeked out keyboard. Another nicety is that the keyboard itself contains the mappings. So if I use the ErgoDox on another machine I’ll get my mappings. Which is not the case if I’m using DVORAK on my normal laptop, but then switch to my wife’s where she’s using the standard QWERTY layout. Thoughts on this?

I realize many just do not care. A keyboard is a keyboard is a keyboard. For those I’m sorry you suffered through this post. For the rest, realize the the keyboard is currently the biggest interface we have with computers. If we can interact more efficiently and ergonomically isn’t that a noble effort?

(the package from Massdrop)

Full Disclosure. I have no affiliation with Massdrop (other than they are located near my Alma mater). They delivered me a great product. They are planning on doing another run of these soon. If you are interested follow them, or follow me on twitter. I’ll probably tweet about any new group buys they do.

"PyCon Preview: Functional Python and Comprehensions"

2013-01-09T23:46:00-07:00

PyCon is right around the corner, and I am lucky to be able to deliver two tutorials this year. For our Utah Python meeting in January, I will be going over a portion of the Hands-on Intermediate Python Tutorial. People seem to enjoy this tutorial, and since I've been trying to read up on Clojure a bit, I thought I'd expand the functional section ever so slightly.

If you are in town, Needle.com is hosting and providing food starting at 6:30. Otherwise you will have to sign up for the tutorial at PyCon. Here are slides from the Functional portion.

Feel free to follow me on twitter, or leave comments. Cheers!

"Advice to My Younger Self"

2012-10-11T06:33:00-06:00

Every time I see an intern or new junior programmer I fight off the urge to give the the advice I wish someone had given me. So rather than sound preachy, I'm going to pretend that I'm from the movie Frequency and give advice to my younger self. In that movie (released in 2000), one of the characters is fore-warned to invest in an oddly named company---Yahoo.

In a similar vein, I would tell my older brother not to just be a designated driver for a fellow student named Sergei, but actually take interest in his research and his novel idea, PageRank. Rather than work for Big Blue, go help Sergei crawl the web, and when I graduate two years later, convince me to do the same. Or have more faith in that Apple recruiting event you went to in 1998. Believe that the network effect of Windows can be shattered.

Those are silly fantasies because hindsight is 20/20. Being in the exact right place at the right time is different than being in the close to right place at the right time. It is hard to predict where lightning strikes. But it is possible to have multiple lightning rods up, rather than insulating yourself from them.

During the last few months of my schooling, life was very exciting. We were in a huge bubble and didn't know it. People were dropping out of school to work for well paying jobs. I even interviewed at a company with the express intent of dropping out with a semester left to work.

During this time a buddy and I had just won the competition for best Senior Project. We had created a colloborative browsing engine, that let you surf the web and chat with your friends in real time about it. And during the day of the show, real VC's actually showed interest our "product". Companies with similar products interviewed us to join their teams, we even pitched VC's (who claimed that our solution being based on MySQL and not Oracle would never work). By having something of value we were in demand.

In the end being disenchanted with VC's we went off to work for companies while intending to moonlight on the project. I went off to work at an "Enterprise" Search company. My buddy was off to Apple. I believe Apple's stock was around $10 at that time. At this point those silly fantasies bother me and I kick myself. They come in and state that some existing companies can be more exciting than startups. I didn't buy it at the time.

Working at a search startup was cool. A lot of my co-workers were PhD types and I loved soaking in their knowledge. Along the way I would slowly learn two axioms of programmers. (Or I should have learned these and wish that someone just sat down and said them):

Programmers are lazy
Programmers want control

Of course programmers are lazy, they make computers do their work for them. They also have a tendency to do the least amount of work possible and once they become comfortable in their situation they are like a raft floating down a big river. Occasionally they need to row hard to change their position, but if the river is cooperating they only need little course correction. It can be a nice little rut to be in, especially if lightning struck you and you decided to take interest in Sergei's project or think that Steve Jobs could actually turn around this floundering company. On the other hand, most startups fail, and a rut can be a tractor beam that leads to the Death Star of complacency or job loss.

Programmers, in addition to being lazy, are prideful, arrogant snobs, who love certain tools and programs for no apparent reasons and love to force others to accept these beliefs. Or they are just lazy and go with the flow. They want to control the search space, or re-create a company that had been reduced to nothing by a north eastern monopoly. Programmers want to reinvent the wheel themselves. Part of this is a desire to understand all aspects of a system. Part of it is arrogance. Harnessing Open Source enables you to lazily control the system.

During this time of the heyday of Slashdot, I was installing Gentoo and playing with Python outside of work. I would read about the ongoings of user groups and the cool things they would do with such technologies. Said user groups happened to be in my backyard, but rather than being an active participant, I would just read about their events after the fact. Perhaps this was due to my introverted nature, or perhaps because I'm a lazy programmer.

During this time the bubble collapsed. My company started going through round after round of layoffs. Perhaps because I was a good engineer (but probably more likely because I was cheap relative to all the PhD's), at every round (I believe there were 7 or so), my job ceased to exist. But (due to my mad skills (or my relative cheapness)), a new opening within the company occured with every layoff. And I took them, going down with the sinking ship. I had officially entered the rut.

Had I put in the extra effort to get involved with the local user groups, I'm convinced that I would have not been in this rut for very long. I've seen many people expand their careers via user groups.

So barring wishful thinking about working at a once in a lifetime company, my advice to my past self would be to get involved earlier. If you put forth a little effort getting involved in your community (and networking at the same time) you will have the opportunity to have more control.

My advice? Get yourself out there. Blog, attend gatherings, tweet (when it is invented), contribute back. Sitting around in your rut only insulates you.

"How to Set Up a Goaltimate Field"

2012-09-04T07:07:00-06:00

I continue to prefer Goaltimate for groups of 13 or less. It's a great game, one gets lots of touches and newbies, seasoned folks can play together. I think it improves new ultimate players throws better than ultimate does. It's great for warming up. We usually play until we have enough for ultimate. People continue to be skeptical before playing, and the majority comment something like this after playing:

That's a lot more fun than I thought it would be

That's cool. Where did you get the hoop?

I prefer this to ultimate

Ok, only a few say the last one, but they have said it. Then there's the group of people who won't give goaltimate a chance. They'll either sit out and wait til there's 10 people and then want to jump to full field 5 on 5 ultimate, or they'll play grudgingly. There seems to be another small group of people who believe that goaltimate is too hard on their bodies (knees and ankles) who also won't play unless they've got 5 other people begging them to. But I'm digressing.

A complaint I've heard is that it takes too long to set up goaltimate. I don't think that's valid. (With the aluminum hoops it's even quicker). A field can be set up in about 2 minutes. Here's how:

How to set up a Goaltimate field

Step 1: Find a good field

A benefit of goaltimate is that it can be played in smaller areas. I've played inside in basketball courts. Can't do that with ultimate. Also, nice large grassy areas tend to get chewed up quickly. But smaller ones not so much. So you can usually play goaltimate on a better field (often it's right next to the ultimate field).

Step 2: Step up the hoop stakes

Because the hoop is fixed in size, it's important that it be set up the correct distance of 18 feet. Here's an easy way to do it. If you got a pvc set (with 8 pieces) connect 4 of them. That will be 16 feet long, so stick a foot on each side and you've pretty much got 18 feet. If you've got the nicer aluminum poles, find the middle and mark it with a sharpie. Here's an illustration that should make it pretty clear.

Setting up the pole is a two person job. (As is taking it down). Have a person on both ends of the pole and put both ends on the stakes at the same time. People seem to be wary of making the pole bend, but it will do it.

Step 3: Goal line cones

The goal line cones are set inline with the hoop, 3 feet away from each stake, making them 24 feet apart from each other.

Step 4: Back goal cone

From the hoop take 8 paces back (I'm assuming your paces are 3 feet), and drop a cone. Note that the depth of the goal is the same as the width of the front cones. It seems that some like shallow endzones (perhaps the intent is to make the goal actually be a semi circle, which it is not). The full endzone opens things up a little and allows for more endzone throw possibilities. Here's an image:(The rest of the images are top down) (Note that the regulation field image (the circular one) doesn't have the endzone scaled correctly)

Step 5: Finish the goal cones

From the back cone I work my way around with 3 more cones on each side.

Continue in the 2/1 pacing around.

One last time

Step 6: Make the "crease" (optional)

If your players tend to goaltend a lot and don't like the 3 second rule (or just step out if front of the hoop and poach there), you can make the crease in front. The rules specify that you can't hang out in the area 1 yard in front of the goal either.

Step 7: Set up clear/2 point line

From here on out I'm no longer "regulation". I've never seen a circular field in the different games I've played at. Most are either rectangular or boundless. When we play indoors, we play rectangular (obviously), and we play without boundaries outside.

Regulation calls for a clear line of 30 yards that arcs out from the hoop. And then a 2 point line (that is parallel to the goal line) at 20 yards out. Again we normally just set up four cones about 20 yards back for a clear/2 point line. The distance of 20 yards can be tweaked depending on energy levels, number of players or field size.

Another (non-regulation) tweak that we add is a cone about 10 feet behind the endzone. This is the "no-clear" area. If the disc ends up behind it on a turnover, you don't need to clear. Some people are confused in that they think they can clear it there as well. NO! If they pick it up behind the cone, then they are cleared, otherwise the only way to clear is the normal clear line.

Here's another illustration with dashed lines indicating what the cones are meant to

Other tweaks

We normally play 3 on 3 or 4 on 4. If we've got more people, then we'll sub on error. What that means is if you throw an uncatcheable throw, or you drop a catcheable throw, you sub out. This tends to give everyone a pretty fair playing time.

I've posted previously about how to make hoops. My original PVC set got a crack once as a player collided with the hoop (must have blended in with the white snow at the time). I'm a fan of the aluminum hoops (which I also have). They are better in every way; lighter, smaller, easier to store, easier to setup, easier to take down, and more durable. Why weren't we using them yesterday you ask? Because my friend who has (permanently borrowed) my PVC set arrived first....

"Real-life Generators and a Peeker too"

2012-06-28T00:20:00-06:00

After a long day of manual labor last weekend, I spent a couple minutes relaxing by converting some PDF's to mobi files so my mom could read them on her Kindle. Her Kindle supports PDF, but reading PDF's on Kindles (especially of the non-DX eink variety) is a pain. You can zoom into sections, but it isn't appropriate for long reading, and the default fonts are too small. One of her favorite features is to bump up the font size on mobi files, especially at night. So I obliged.

To that end I've created some code that helps in this process---ebookgenerators. Most readers don't care about the process of converting pdf to mobi. How I cleaned up the text might be interesting though. I used a chain of Python generators!

A few have complained that my Iteration and Generator book doesn't have enough real examples. Some form of this blog post will probably end up as a chapter there. So without further ado, here's some code.

A concept I briefly mention in my book is a Peeker class. A peeker can look ahead during iteration. This is useful if deciding the output of an action requires more than one item. Here's mine:

class PeekDone(Exception):
    pass


class Peeker(object):
    def __init__(self, seq):
        self.seq = iter(seq)
        self.buffer = []

    def pop(self):
        if self.buffer:
            return self.buffer.pop(0)

    def peek(self, n=0):
        """ this can raise an exception if peeking off the end. be
        aware and handle PeekDone appropriately"""
        try:
            if n == len(self.buffer):
                self.buffer.append(self.seq.next())
        except StopIteration as e:
            raise PeekDone('Exhausted')
        return self.buffer[n]

    def __iter__(self):
         return self

    def next(self):
        if self.buffer:
            return self.buffer.pop(0)
        else:
            return self.seq.next()

I use the PeekDone exception as a sentinel value, rather that returning a special value. Here's an example of a generator removing double blank lines from lines of text using Peeker:

def remove_double_returns(lines):
    lines = Peeker(lines)
    for line in lines:
        try:
            next_line = lines.peek()
        except PeekDone as e:
            yield line
            return

        if blank(next_line):
            yield line
            lines.pop()

        else:
            yield line

That could be done by someone fluent in awk in probably two lines. But here's one that I wouldn't want to touch.

def fix_space_in_paragraph(lines):
    """ If paragraphs span pages (often) then there could be extra
    returns in the paragraphs....
    """
    lines = Peeker(lines)
    for line in lines:
        try:
            line2 = lines.peek()
        except PeekDone as e:
            yield line
            return
        try:
            line3 = lines.peek(1)
        except PeekDone as e:
            yield line
            yield line2
            return
        if blank(line2) and (not ends_sentence(line)):
            # don't use line2 so pop it
            lines.pop()
        yield line

Here's a simple generator without Peeker. I need to ensure that paragraphs have a empty line between them so docutils does the right thing:

def insert_extra_paragraph_line(lines):
    for line in lines:
        if ends_paragraph(line):
            yield line
            yield '\n'
        else:
            yield line

In the end, using a chain of these generators, I was able to generate three mini-ebooks for my mother before she left for a week-long cruise.

My scripts for cleaning up the text looked something like this:

import sys

import ebookgen


def run():
    data = sys.stdin
    data = ebookgen.remove_leading_space(data)
    data = ebookgen.remove_dash_page(data)
    data = ebookgen.remove_carot_l(data)
    data = ebookgen.remove_two_spaces(data)
    data = ebookgen.remove_double_returns(data)
    data = ebookgen.insert_extra_paragraph_line(data)
    data = ebookgen.insert_rst_sections(data)

    for line in data:
        print line,

if __name__ == '__main__':
    run()

"I Would Like Sphinx PDF Generation to Support the LaTeX Memoir Package"

2012-05-17T06:11:00-06:00

I would really like to use the same toolset for publishing POD (Print On Demand) books that I use for ebooks. Yet the pdf generation abilities of both docutils and Sphinx seem more focused on pdf "ebooks"---pdf's that you would read from a computer---not pdf's that are approriate to create physical books from. I think the solution is to add support for the LaTeX "memoir" package to Sphinx. I'll eventually get around to it, but it is not on the top of my stack. To that end I'll pay up to $200 (paypal or Amazon gift) to get that support.

For those not familiar with memoir, here's a snippet of the README:

Memoir is a flexible class for typesetting poetry, fiction, non-fiction and mathematical works as books, reports, articles or manuscripts. Documents can use 9pt, 10pt, 11pt, 12pt, 14pt, or 17pt as the normal fontsize and, if you have scalable fonts, 20pt, 25pt, 30pt, 36pt, 48pt, or 60pt sizes, or even larger. Many methods are provided to let you create your particular design. The class incorporates over 30 of the more popular packages.

Here are the details. The first person who notifies me of two github repos that meet the following points will get $100:

A github repo containing a Sphinx project that renders out to pdf using the memoir package. The document must have an example of:
Half Title Page
Title Page
Copyright Page (on verso page)
Dedication
Table of Contents
Introduction
Chapters (should start on right page)
An index
blank pages at the end
Customizing page size
The chapers should have an example of:
- Text spanning multiple pages (lorem ipsum is fine)
- Normal text
- Italicized text
- bold text
- monospace text
- Sections
- footnotes
- a code block
- an admonition
- a long admonition that is longer than a page
- an admonition with a code block inside of it
- a figure
- a table
A github repo containing a fork of Sphinx that has memoir support. When using this fork with the above project, a suitable pdf should fall out. Numbering of chapters is correct, page numbering is correct (odd on right side), index is correct, etc.

Ideally most of the book (Table of Contents through Index) would be pure rst (or sphinx rst++). I'm ok with raw directives on the front matter.

If you are familiar with LaTeX and Sphinx, this should be pretty straightforward. It would also be a great project for someone interesting in learning those technologies.

The person who can deliver the above will make me quite happy. But, as I said, I would pay up to $200. The remaining $100 will be appropriated to the developer (or someone else) who sees that this gets pushed upstream into Sphinx proper (ie when I see a bitbucket commit with the changes on the Sphinx project). So the person who files the bug, patch and follows through with any needed tweaks to get this into mainline will get that money. I reserve the right to split up this portion among people as well.

(I apologize in advance for the github requirement, I prefer that system, so the first deliverable needs to be there. Given that Sphinx prefers bitbucket, you need to play well with them for the second deliverable.)

"Coding Is a Life Skill That Should Be Taught in Elementary School"

2012-05-16T21:04:00-06:00

As a seventh grader I had the choice between learning French and Spanish. I choose Spanish because it was practical (or at least my parents convinced me of that), while it seemed that all the cool kids took French. Excuse my French but I think my parents were right---taking French expecting to actually use it (or have a reason to use it) would be a waste of time. (Sorry to the 2% of my blog visitors to whom that hits home). In fact the same could be said of the two years I took Spanish. I remembered only the colors and my Spanish name, "Paco".

But taking Spanish for two years twisted my brain enough that I actually took a year of German after (again useless if I were expecting to use it to purchase say a Mercedes). And it was easier this time. When I did become fluent in Spanish three years later, I was grateful that I had learned how to conjugate verbs previously. My uptake was quicker because I had practice. In two months I was dropped off in the middle of Colombia and held my ground ok. (Later my Spanish professor would inform me that speaking in Spanish is insulting in Gringolandia, and I don't pull it out too much now. So it was useless.)

Likewise, the Pascal class I took in High School wasn't super practical. I didn't even have a machine at home to code Pascal on. I haven't coded in Pascal since. Yet the logical manner it taught me to think has been a wonderful life skill. It has even helped me to debug plumbing issues. When a the faucet is clogged, I could just disconnect the pipes connecting the faucet to the wall and start snaking around in the wall. But (given that I have girls around with long hair) it makes sense to start right at the faucet itself and snaking that without even disconnecting any pipes. Less mess, and more time for other less grimey things.

(Alas learning more about plumbing has helped in other areas as well, such as knowing how to replace a toilet, or stop a leak. It's probably a decent life skill to have if you are a homeowner or live in a place with running water.)

The section I took on Lisp in college was not super practical either. Even then no one programmed in Lisp. Why did I waste that time? Because I have experienced functional constructs, I now appreciate them and see their utility (and even the allure of Clojure). I appreciate list comprehensions, first class functions and the laziness of generators. (If you have read this far in the post you'll notice that I also like parentheticals.)

I had the chance to teach ~80 year old man how to program. He wanted to learn just for the sake of understanding what the little creatures inside of his computer were doing. Waste of time?

I've taught 3rd graders how to create ebooks. That was really an excuse to teach them xml, html, css, typography, fibonacci sequences, image compression, debugging and actually learning how to use a computer for something other than clicking the blue "e" and typing in starfall.com. I neglected to inform them that they could have just as easily written their books in Word and exported to HTML. I don't feel guilty about that. (Some even wrote thank you cards in both HTML and the rendered output!)

I have also been able to teach 3rd graders programming. They were able to create simple text games with conditionals, functions and looping constructs. Did they create the next 3D shooter? Did they even create a game that they'd like to play more than once? Mostly not. But they learned a little bit about how computers work. They also learned how to think logically, and how to be specific. (My wife compiler still complains that my English isn't specific enough---too many "this" and "that"s at line 0. To which her compiler responds "I have no idea what you are talking about!")

Today, the world still runs on Excel. A poor man's database or programming environment. Just today a co-worker asked to help move some data from Excel to a program so more advanced calculations could be performed on the data. There was also an urgency to get rid of the manual labor of tweaking rows and columns. The end user wanted automation and power. (And to be lazy).

If you have any inclination of working with programmers you should take CS101. You need to have some semblance of understanding of what they need and what you need to help them. Do you feel like a doofus when a car repair person is telling you what you need done to your car? Do you want to get ripped off by the plumber telling you the twenty things he needs to fix?

How often am I burdened by my (loving) family and neighbors, who knowing that I "work with computers" think that that means I can clean the spyware off their XP installation! I imagine that I would not play that beast of burden if others had some understanding of how computers and the internet actually worked.

Surely a little understanding of what powers the digital aspects of our lives can't be too harmful.

"Utah Open Source Conference Python Tutorial Materials"

2012-05-04T21:50:00-06:00

Assignments for UTOS Hands-on Tutorials

"Indy Python Workshop and UTOS Conference"

2012-04-13T07:10:00-06:00

Today and tomorrow the Indianapolis Python Workshop is running. From the website, "The Indianapolis Python Workshop for women and their friends is a free, hands-on, beginners' introduction to computer programming." This event was inspired by a similar event in Boston.

I love this local effort to empower others by teaching Python. To show my support for the event, I'm running a promo on my Treading on Python. It will be discounted to $0 during the event as supplementary material.

For people local to the Rockies, the Utah Open Source Conference is coming up next month. In addition to many awesome talks, I'll be giving 2 three hour workshops on Beginning and Intermediate Python. Though not free, the low cost of the event is a great deal considering the quality and quantity of the presentations.

"Treading on Python Promotion"

2012-03-27T07:07:00-06:00

Treading on Python will be on promotion today. If you are a Kindle owner and interested in learning Python, today is your lucky day. If you prefer handouts/cheatsheets here's another freebie. Cheers.

"KDP Select for Technical Non-fiction Books Month 2"

2012-03-26T06:42:00-06:00

This post describes my efforts at using Amazon KDP select to sell technical non-fiction books. After some success in my efforts last month (reaching the top of my sub niche), I can report that I fell back a few spots but for the most part sales have been steady. As can been seen in my Amazon sales chart, the rank for my Treading on Python book have been semi flat for the past month. Sales of my Decorator Book have been similarly flat. They have been hovering in rank 8-15 which is pretty good considering the other books are the standard Python recommended books or a textbook. Apparently self published books can do ok, considering my books and that number 1 or 2 is usually Al Sweigart's Invent Your Own Computer Games With Python.

What has happened in the past month?

During PyCon I ran a promotion on Treading on Python. The Rasberry Pi folks picked it up (probably due to Steve Holden's tweet) and I actually had more sales in the UK that in the US that day. (Odd considering that US is around 8-10X sales of UK in my experience.) Given that sales have been relatively flat, I'm wondering if I have peaked. Perhaps I have. Though I struggle with efforts to promotion. It seems I'm labelled as a troll if I try to post anything to reddit programming or hacker news related to my book. Which I find a little odd, because others seem to do so with ease. I can promote through Python channels, which for my more advanced books seems to work, but Treading on Python is aimed at people wanting to learn Python, so promoting through Python channels seems a little less effective there.

I'm still going to be running a few more promotions for my books. They seemed to have been effective.

Another challenge has been getting reviews. I think every author struggles with that. (And reviewing is long tail behavior).

In the meantime I've also silently pushed out my next guide, Guide to: Learning Iteration and Generators in Python. This book is not in KDP Select. I'll probably put a bundle back up on my site and perhaps BN or Kobo. If you are interested in a review copy let me know.

To close off this post. I think technical self publishing can be "successful". I'm not sure that you will retire like fiction writers who hit the jackpot. But I'm pretty sure that with Amazon's 70% (or more if I sell myself), I'm getting almost as much money as people selling their books for $20-$40 (most of the rest of the people in the Python top 20 list).

"PyCon 2012 Review"

2012-03-12T06:35:00-06:00

Here's just a few notes from my PyCon 2012 experience. This was my 7th PyCon, and though the conference has changed over the years, I think it is mostly for the better. A general theme of PyCon this year was data analysis. I'm probably biased since I'm doing that at work. In contrast with previous years, where there seemed to be two Django tracks running concurrently.

The quality of the talks has also increased over the years. Most of the talks I attended appeared to be well rehearsed and pretty well thought out. Though I did hear some grumbling that many talks could have gone a little deeper.

The quality of the food was excellent this year. The quantity less so. I heard one attendee claim he "lost 5 pounds during the conference". But herding 2200+ people through the lunch line is no small task and the team did it with efficiency and ease.

The layout of the conference was a little confusing. There was a double indirection between talk locations and the actual locations. Talks were located in a track. A track was located in (what appeared to be an arbitrary) room number. Said room number did not line up with the pre-existing hardcoded conference room numbers and in some cases conflicted. This caused confusion on the first days of both tutorials and sessions. With 2200+ people confusion of such type can be annoying.

Also the layout of Open Spaces, and the open space signup board was relegated to a far side of the building that didn't get much traffic. The two BoFs I hosted were poorly attended. Perhaps that is due to marketing or the subject in general (yes, Python tiling wm's are boring), but there was very small traffic through the open space, with the exception of the TiP BoF. This BoF was probably larger than the first PyCon I attended.

Previous years also had a job board. I told a friend that I would post a job listing for them. But alas, I never saw the job board this year. The exhibition hall was perhaps a huge t-shirt dispensing machine, but no corkboard to post a description on. If you are interested in changing the world working at a non-profit, ck12.org is looking to hire.

The PyCon 5k was a excellent respite from days of sitting and breathing regurgitated air. The amount of minimalist and even barefoot shoes relative to the general population leads me to believe that pythonistas are geeky, enlightened or suffer from Stockholm Syndrome. I really enjoyed it. During my final tutorial an attendee pulled out an ultimate disc. It was great to get outside in the wonderful weather and toss the disc around.

My tutorials were packed. My biggest issue was the influx of people 25 minutes into the first session. Being that it was beginning Python, that caused a small delay as I helped people get going. It is pretty satisfying to walk around during the assignment portion helping people out and realizing that 95% of the people are busy hacking away. They weren't just browsing Facebook or reading email, they were actively learning Python. One attendee told me it was the best tutorial he had ever attended, which is nice to hear. I also recieved some more constructive criticism from others. I'm always open to feedback, and if you attended one of my sessions you should fill out the tutorial survey.

Yes, Python videos are now up on the web. Some of the content is there. But the conference is more that the videos. I was talking to someone who was relating that they had an issue with translation while they were travelling in Siberia. They did a lazy twitter and got a response. While we were talking about this a fellow entered the conversation who was from Russia. Lo and behold he was the lazy web helper, but the two had NEVER previously met in real life before.

So yes, PyCon is about the people. And even though there were a lot this year, it was a great conference.

"Not at PyCon? Learn Python Anyway"

2012-03-07T07:36:00-07:00

PyCon starts today. It's great to be back in what used to be my backyard for PyCon. I love the Bay Area. To celebrate I'm doing an Amazon KDP Select Promotion on my Treading on Python book. During an Amazon "promotion", the price is reduced to $0.00 for the day!

This book is aimed at getting programmers quickly bootstrapped on Python and should correspond quite well with my Beginning Hands-On Tutorial (go figure). Now I'm off to register and teach for the day.

If you are more of the handout/cheatsheet type, you might enjoy the handout for the tutorial.

Feedback as always is welcome. If you are at PyCon say hi. Cheers.

"PyCon Hands-on Assignments"

2012-03-06T22:09:00-07:00

Assigments for PyCon Hands-on Tutorials

"Handout for PyCon 2012 Hands-On Python Tutorial"

2012-03-06T08:01:00-07:00

Here is the latest iteration of my beginning Python handout. It has gone through a few iterations over the years. It is pretty cool that about 90% of the basics of Python fits on both sides of a legal sized paper.

"Amazon KDP Select Non-Fiction Week 3"

2012-02-20T21:54:00-07:00

Yesterday was the start of the ultimate test of the promise of Amazon KDP Select. The Kindle Direct Publishing Select program is a 90 day exclusivity commitment to Amazon, in return for some reward. Amazon reports authors finding ~30% increased sales due to enrollment. I've read a bunch of fiction author's reports, but none for non-fiction, so I figured it would be a nice experiment. I was previously selling on Barnes and Noble as well as a Epub/Mobi/Pdf bundle on my own site. BN was pretty weak, but my bundle was close to 30%. I figured I could turn it off for a while to experiment.

I ran a "promotion" on one of my books. A promotion for those who are not familiar with Amazon KDP Select is a day where the book is discounted to a price of $0.00. You are allowed five of them when you sign up with Amazon KDP Select. Yes, for the honor of being exclusive to Amazon for 90 days, they give you the ability to sell your book for nothing.

The idea of a promotion is not new. The idea of free promotion in this world of social media is a little different. I have seen first hand what happens to iPhone apps when they are promoted on such sites. The ideal situation is free loaders (who would normally not buy your digital goods) come in droves and create such a media frenzy that over the next couple of days, those "lost sales" turn into marketing gold and you are riding the waves to being the next Indie Developer/Author retiree. Though staying in that short head where the few people who actually make decent money is difficult.

I have also been on the other side of this. I will admit, I'm a free loader for Android apps. I visit the Amazon Andriod Market every day just to see what is for free. If the app has good ratings and I think it will be useful now --- or in the future (since I get free updates), I will "buy" it. "Buying it" doesn't force me to actually waste space and download it to one of my android devices. And I'm a very "long tail user" here. I'm not engaged. I don't review or rate the apps. Most of them I haven't even bothered to download. But for the chance that I might it the future, I'll put it in my Amazon virtual app locker. These days my Kindle Fire is mostly used for Carcassonne (an app that isn't even available in Amazon's market), though the kids love the normal apps, Angry Birds, Plants vs Zombies, and Where's the Water (all Amazon "purchases").

I can see the appeal of this for fiction where you have potentially mass appeal. I'm not sure it holds water for non-fiction. Right now my Guide to: Learning Python Decorators is #1 in Programming and #275 overall in the "Free" Kindle Store. It "sold" about 13X more copies during the promotion than it did the other 19 days of the month. It is also possible that I just gave the book away to 90% of the people who might be interested in it. I'm hoping that maybe I'll get a positive review or two out of it. The feedback I've gotten previously from my books has been positive. But I think that the crowd who purchased it was very self selecting. When you discount the price to a freebie, the net widens. It will be interesting to see if the net catches many high value short head readers or not.

The Good

I actually received useful feedback from online comments. My book is more of an introductory treatise on its subject and does not cover minutiae and corner cases that most people do not run into. I knew this and have two thoughts. One --- I'm in the process considering how to integrate that content in a way that I like. Two --- said commentor is not in the target audience of this book, they already live this stuff. But that is the hand I am being dealt. I think to "market" my book I need endorsement from "luminaries". Yet, they are usually going to endorse something that is appropriate to their level. With my book they are going to pick over every detail, not realizing that the basics that they've lived and breathed for years are confusing and difficult for many to understand.

I'm still working on creating a physical version of my books. I'm getting closer. I'm messing with margin sizes (I like to jot in my non-fiction physical book margins) and formatting of front matter.

I can't complain that I'm basically in the top position for my genre. Isn't that the dream of any self-pubbed writer? I'm not sure how long it will last. And I'm curious to see the long term effects of this "promotion".

The Bad

I did limited marketing. I was away skiing today, so I blogged the night before and tweeted before hitting the slopes. I tried getting it on programming.reddit.com, but am labelled as a troll there, even when I try to give my book away for free. This is somewhat annoying, because any marketing to what I think comprises my target audience is automatically moderated as spam.

On the flipside, python.reddit.com picked up the book. Sadly they linked to the Amazon page rather that my blog or the book's page. (Perhaps that is why prog.reddit moderated down, because of my affliate links ... strange because I get $0 from a $0 sales but I digress).

Authors often talk about "platforms", which means a different thing for programmers. But author "platforms" are the eco-systems around the online presence of their book. They mainly consist of:

a book home page
a blog
a place to collect email addresses

I would have preferred that the links during my promotion went to my "platform" instead of Amazon's. At least they would have understood the backstory and my reasoning behind the promotion in the first place. I can verify that they did not hit mine because my analytics numbers are less than my sales counts. Annoying, but that is the cost of having socially moderated marketing.

If you are reading this and have less "trolly" ways for me to tell you that my book is discounted to a "free" price (which includes free lifetime upgrades), pray tell in the comments.

People don't like Amazon, Kindles or Kindle readers (or don't have access to them). I received requests asking if I would give them a free pdf of my book. That won't happen while I'm enrolled in KDP Select, being that it is a violation of my contract. Geeks being geeks, they are already discussing the methods to pdf creation, which look sane. I'd like to further explore this segment of potential "clients". Do they really love pdfs? If so why? Are they ignorant of the many available readers? Are they afraid of being tied into the ebook strangle-hold that Amazon appears to have? Again, I'm not sure fiction readers care or are this technical, or that their content is important enough that it needs to be in a pdf format for some reason. But these are some of the hurdles that non-fiction writers may have to deal with.

The Ugly

I also found out this week that some people feel that my book has limited value (I hoped they picked it up when it was priced accordingly). It is not a sentiment that I am entirely unfamiliar with. Having worked very closely with open source (not just as a free time thing, but actually being employed to do so), I'm aware of "information wants to be free" arguments as well as "people associating non-zero cost with value". Many outfits run open source backed by paid solutions to enhance performance, be they software or hardware. But occasionally you rub someone the wrong way.

So I'll say it again. Most of what my books deal with is information that can be found on the web. I've collected, curated, and refined it to a more palatable format (I hope). As do most programming authors. My content has consistently been one of PyCon's most popular tutorials. I have no problem with people learning from freely available sources. But many times I would have preferred a well written book on a topic, rather than trying to piece the puzzle together with a bunch of disparate sites.

Other people have actually written me, thanking me for a concise guide type format to a specific idea. One actually listed a bunch of potential topics. I also have topics that I would be willing to pay for, but am in no way qualified to write.

I'm wondering if price has something to do with it the gripes. Yet most programming books are expensive and not read. Even more so if they are college "texts". Mine are relatively shorter, cheaper, and hopefully easy to read. Maybe I'm making a mountain out of a molehill here.

The tools that Amazon provides for discounts are pretty limited. One nicety of selling my own bundle was the ability to provide discount codes. I understand why the limitation is there for physical goods, but am not sure why they do not support this for virtual goods. I have some ideas, but have heard nothing concrete other than, "if you want to discount, you do a blanket promotion and give it away free to everyone".

Other notes

US sales ~20X UK, ~40X DE, ~100X IT, ~160X FR, ~400X ES. Sadly I don't have website numbers to correlate with Amazon sales due to direct linking to the book in Amazon.

"President's Day Decorator Sale"

2012-02-19T22:38:00-07:00

The Grand Rapids Python User Group meets on holidays. They meet on holidays to discuss decorators (among other things). They also email authors of Python decorator books and ask them if they have free copies to give away at their meetings. Given that I'm subject to the terms of Amazon KDP Select, I currently cannot give away an individual copy. But I can use a "promotional" day to lower the price to $0 for Feb 20, 2012. In return I'm now an official sponsor of the group!

Enjoy. Tweet it, reddit it, etc. If you like it, or find it useful, reviews on Amazon or blogs are always appreciated. I am off to the slopes for the day so there might be some radio silence on my end.

PS --- I'm still working on getting a dead tree version and am closer than I was last week. Also, I'm hoping to release the next "Guide to: Learning ..." this week. Stay tuned.

"Decorate Your Shirt"

2012-02-17T06:42:00-07:00

I messed around with Cafepress last night creating a shirt that I will probably use as a prize at my PyCon Intermediate Hands-On Tutorial. It has a pretty good decorator on the back.

BTW --- Someone should tell Cafepress about this technology called Ajax...

"Amazon KDP Non-Fiction Week 2 Update"

2012-02-13T06:03:00-07:00

I am two weeks into my Amazon KDP non-fiction experiment. Guide to: Learning Python Decorators is currently #15 on the Python list as I write this, but was up at 5 during the week. It is aso #1 on the Python "Hot New Releases", which surprises me because #2 is a really strong title. Sales of both of my books have been slow and steady. I heard that the average "indie" author only sells 4 books/month. I'm happy to report that I'm slightly better than that and I will be receiving a check from Amazon this month ($100+ in sales).

What else have I done this week? I have wondered about Amazon's lack of existence in BRIC (Brazil, Russia, India and China). When I had my own store front, I was able to sell to a few individuals from some of those countries. Now I am not because Amazon has no presence there. Again this is probably not something Indie fiction writers care about (with perhaps the exception of India) — I assume fiction is somewhat language dependent. But many programmers know English, so presumably I am losing some sales. How big would they be? I am not sure, but I have received at least one request from someone who claimed they did not have Kindle reader access. Currently my non-US sales are less than 10% of US sales. I am not sure if that is a function of my marketing, or how strong the Amazon presence is in those countries (UK, DE, ES, IT, FR), or some combination of the two.

I also signed up for Amazon Author Central. Perhaps this will help with SEO, I am not sure, but I now have an author page, which also aggregates this blog and my tweetstream.

While at the Utah Python group this week, the first presenter actually mentioned my decorator book in his talk. When the meeting ended, an attendee informed me that he thought my book was pretty good (which he had purchased and was reading during the second presentation). Also he gave me some more feedback, which is always nice.

I also spent some time looking into pbook generation. My calculation is around 4% of people would prefer dead tree books (very unscientific, consists of requests divided by current sales at some point in time). KDP only requires exclusive digital rights to your sales, not physical, so this is a very real possibility. But, my current pdf generation has issues and I starting learning LaTex to solve those. I hope to have some sort of dead tree versions released soon. I'm thinking about using CreateSpace, but would be open to using other POD services. If you have experience with POD feel free to drop me some notes.

My book was submitted to the January 2012 Ebook Cover Design Awards, which was released this week. I was hoping to get some feedback on my cover, but did not. I interpret that response as an indication of the “blah”-ness of my cover. Not sufficiently interesting to warrant any praise, nor blatantly bad enough to expose the horrible design elements.

Finally, a huge advantage to KDP, is I can easily push updates and Amazon handles everything else. A disadvantage is anonymity of purchases. With sales from my site, I had emails for all purchases and could interact with buyers, but pushing updates was a bit more of a pain. Some food for thought if you are considering one or the other. I really like to think that ebooks are somehow “alive” and that I can incorporate reader feedback very quickly and actually push it out, though this is not something that “real” publishers seem to be doing yet. I'm thinking that I might stay on KDP with my more expensive/longer Treading on Python which receives a higher percentage of "rentals", and make distribution of smaller ebooks more widely available — Kobo, my own site, and perhaps BN again.

"Utah Python Feb 2012"

2012-02-08T06:10:00-07:00

The Utah Python group will be meeting on Thursday, Feb 9th at 7pm. Amji will be doing a short presentation on a memoization decorator and Eric will be giving a preview of his PyCon talk "Interfaces and Python". Cheers.

"Amazon KDP Week 1 Update"

2012-02-06T22:23:00-07:00

I am one week into the Amazon KDP non-fiction experiment with my books. Guide to: Learning Python Decorators is currently #12 on the Python list.

My sales have already outpaced any Barnes and Noble sales I've had. (Though I'm wondering how much reach Kobo would give me for the international market). What promotion have I done so far?

A few tweets
Wrote a (buggy) cheatsheet for Python decorators that got linked to Python section of reddit
Updated buggy cheatsheet
Got a wonderful review for my decorator book on Amazon. Thanks Richard! Money can't buy such nice reviews

From looking at my analytics, reddit gave me about 300 hits, twitter another 100 and google about 50. This resulted in 14 sales and 6 lends. Not bad.

Also looking through the list of non-fiction Python books, most are $30+. My decorator book is pretty short (a couple of chapters and priced at $2.99), but my Treading on Python book is about 140 pages. The way Amazon pricing structure works, I'm penalized if I price between $10 to $20 (above $9.99 and you are in the 30% royalty system rather than 70%). Perhaps this makes the shorter guide type books more reasonable and a "no-brainer" purchase. But I'm wondering if the lower price hurts with my longer non-fiction book? Do people equate price with quality or density of useful information?

"Learning Python Decorators Cheatsheet"

2012-02-04T08:39:00-07:00

Decorators are useful in Python for a variety of reasons. Some common uses are:

caching
logging
timing
authorization
and lots more …

I mentioned in my recent SCALE10x talk that I would have a handout/cheatsheet for Python decorators. Here is a pdf version of the Decorator cheatsheet. It has templates for normal decorators and parameterized decorators. Enjoy!

edit — I apologize for the typos uploaded previously. I ran the handout through doctest and it should be good to go now.

"Amazon KDP Select for Non-fiction Technical Books"

2012-01-30T22:18:00-07:00

tl;dr

I'm experimenting with KDP for niche technical programming books.

The Marathon Begins

About a year ago I decided to write a book. I didn't have a story to tell. I wasn't planning on making money. My impetus was to prepare for a 3 hour long conference tutorial I was giving. Rather than spend my prepartion time creating slides and going over them, I figured I'd write a book covering the material. It worked out pretty well. Except the book's release was delayed by about 5 months. Yeah, writing a book takes a LOT more work than I would have thought. In the mean time I drafted "the Python book I wanted when learning Python" and made it available as an epub, which I put on Barnes and Noble, a mobi which was sold on Amazon and a bundle containing the two e-books formats as well as a pdf that I sold on my site.

With the release of the Kindle Lending Library, where Prime members can borrow a book per month, there have been reports of how KDP can boost sales. These posts seem aimed at fiction (as does most of the self publishing world). There aren't any reports of technical writers making tons of sales (barring 37 signals). So I'm trying experimenting.

KDP Select for Technical Books

Beginning 10 minutes ago, both of my books are only available via Amazon. I'll be reporting back what happens. I'm not expecting a miracle, nor am I expecting to quit my day job. I just want to see if the Lending Library expands my reach.

If you are in a similar situation please feel free to comment below. If you are looking for an epub or pdf, sorry, for at least 90 days I'm exclusive to Amazon.

Feel free to follow me on twitter for progress.

"Sideloading Android Market Apps Onto the Kindle Fire"

2011-12-31T05:35:00-07:00

I have the game Carcassonne on my G2 which I purchased at the Android Market. While there are a few things I like about the iOS version better, I think the Android version is a well done game. No, they aren't developed by the same developer, though gameplay is very similar.

In addition to being cheaper than the physical version, there are other advantages to the digital version, such as indicating valid spots to play pieces. This is a huge benefit and makes the game smoother and faster. This is also a board game, where you don't have private information (such as cards in your hands), so it is very amenable to tablet computing. It seems like most Carcassonne people I know (a handful) prefer the tablet version. But enough about Carcassonne.

Enter the Fire

I wanted to load the game onto my Kindle Fire. Sadly Carcassonne is missing from the Amazon store, but getting it onto the Fire wasn't too hard. Here's what I did, you might be able to shorten some steps.

Purchase the app on my phone.
Enable Installation of Applications from Unknown Sources in the Settings->More->Device page on the Fire.
Create an .apk backup of the game on my phone. The file explorer ES File Explorer, available in the Amazon market, allows you to do this.
Copy the .apk from the phone to the Fire. I used a usb cable to copy from the phone to my computer and the app WIFI Explorer to put it on the Fire. Using Dropbox on both devices would probably eliminate need for a computer. My .apk was located in /mnt/sdcard/backups/apps/Carcassone_5.apk.
Tap on the .apk on the Fire from a file explorer to install it.

Enjoy the Fire

The game works fine on my Fire. Because the Fire lacks some features my phone has, some applications won't "just work". I have a few other apps that I want to transfer over, and feel that this is a viable solution. Here are the drawbacks to sideloading non-amazon apps:

The icon resolution is lower than other Fire apps, so it appears a little pixelated in the carousel.
Ease of install is gone.
Ease of updating is gone.

"Humbug to Poor eBooks"

2011-12-19T22:06:00-07:00

In an effort to remedy only reading to the younger kids, my oldest and I began reading Dickens' A Christmas Carol on the Kindle Fire about two weeks ago. It has been nice to read together and it is really the first book I've read on the Kindle. (I've read my book multiple times, but that doesn't count).

We'd take turns reading paragraphs or pages, and loved that we could read it in the dark or in front of the fireplace. Also the built-in dictionary has been really helpful for some of the archaic language of the book. I love how the definition pops up on the screen so you don't lose your place. Though I admit it is missing some terms, perhaps this eBook would benefit from a glossary.

Plus the book, being as old as it is, is in the public domain so it is free. What could go wrong?

Well, the version we had had this annoying feature of including text such as

Original manuscript of Page 65.

Somewhat annoying. The mechanical turk who transcribed it was pedantic about keeping track of pages, but the final editor seemed to not mind. But I guess you get what you pay for. Also, my version claimed to have the illustrations found in the 1843 original by John Leech. Not true! Apparently the other version available in the Amazon bookstore was some sort of abridgement (or I'm too gullible and read too many reviews on Amazon).

What to do? I figured I had created a framework, rst2epub2, that will not only create a nice epub, but also a kindle friendly mobi file. Here was another chance to use it. So I found a version on gutenberg complete with Leech's images. I fired up emacs and about 45 minutes later I had a lovely version of the A Christmas Carol complete with the original cover and illustrations.

So Bah Humbug, or Merry Christmas to you all. Here's an epub and mobi of a prettier version. Now I just need an app that makes sounds of old crusty page turns and I'll be good to go.

"eBooks for 3rd Graders"

2011-12-13T06:26:00-07:00

I learned so much about serif, san-serif, html and css.
Hannah, Student in eBooks for Elementary Students

I recently had the opportunity to teach a handful of elementary students about eBooks. Every year my daughter's school has what they call "Type 2" classes, where parents come in and teach about subjects for 8 weeks. There are a bunch of pre-canned lessons:

Debate
Engineering
Geography Quiz

Last year I taught the kids Python, and because I was in the middle of trying to get my book, Treading on Python, out for both Amazon and Barnes and Noble I thought that it might be interesting to teach digital publishing. Basically I would teach them HTML with a few sprinklings of typrography, math and computer science.

I ended up covering a few subjects:

History of books
Difference between publishing digital and physical books
HTML
Tags
Well-formedness
CSS
Counting by binary and hex
Discussion of copyright and image re-use
Basic image encoding for gif and jpeg
Basic typography
Serifs vs san-serifs
Ligatures
Ratios - The golden number
Font sizes - double twisted helix
Use of Explorer to show file extensions
Editing HTML and CSS with notepad

I decided not to tell them you can basically use Smashwords in combination with Word to create eBooks. That would have been easy, since they are all well versed in Word. But in these days of simple multi-touch phones and tablets I was amazed at the amount of time I would need to spend with basic computer skills. They understood well-formed HTML quickly. But copying a file, or opening it with Notepad was a novel idea. I'm pretty sure that most "computer time" at school is spent playing Flash games or typing in Word.

<h1>Thank you!</h1>
Melanie, Student in eBooks for Elementary Students who now inlines html tags in her thank you letters

How many of the books are best sellers on Amazon? Well, zero are up there right now. Two students actually "finished" their books, which really consisted of HTML pages. I'm going convert one to epub and mobi so he can throw it on a device. Two other students seemed quite distracted, they were also the youngest so maybe they were intimidated, but they got the basics and just goofed off. The rest were pretty heads down in writing a story in Notepad using HTML and CSS. It was impressive. They were enclosing all of their paragrahs in tags, tweaking the css, using headers and images. And they even survived having IE as the sole renderer of their work. No firebug goodness to help out.

While they didn't walk away with Amazon best sellers, they did learn some basic computer skills and I think they had fun. I've included a few of the cards they sent me when the class was done. Notice the one on the top that has both the HTML rendered as well as the source for the HTML and CSS. Awesome!

Some of my class material is available at my github account.

"Introducing Treading on Python"

2011-11-21T06:49:00-07:00

For the past few months life has been somewhat hectic. Real work has been busy as we have been pushing to release our next generation product. In the meantime, I have been writing a book outside of work. The book, Treading on Python, is a quick paced introduction to Python. It has been an interesting experience, so I figured I would share a bit about it.

Another Python book? Why?

Why would I create another book in the Python space? I have always liked teaching others. Possibly it is in my blood, my mom is a teacher, and I have found it fulfilling to help educate others. Since learning Python myself, I have been able to help many, including one on one with a teenager and, an old retired man, as well as classes and workshops with programming professionals, and elementary children learn Python. This book is aimed at professionals who want to be able to read and write Python quickly.

I have been approached previously to write a book, and I did not feel like I could commit to a schedule while having full time work, and other obligations. But I was intrigued with writing a book. When I had to opportunity to give a three hour tutorial on Beginning Python that I had previously prepared, I decided to spend my preparation time (mornings and evenings) creating a book from the tutorial. More specifically, creating an ebook, where I controlled almost everything. Thus began the journey.

There are many books on Python that already exist. Some are good, some less so, and others I have not read. Some are thick tomes, few are succint. I felt that I could fill in the niche for a quick introduction to Python. If for nothing else, writing the book gave a me a chance to:

Compose my thoughts about the tutorial
Read The Elements of Typographic Style
Learn about ebooks
Learn about the mobi format
Learn about the epub format
Learn about ebook cover design
Learn about selling items directly from a blog
Learn about Amazon KDP platform
Learn about Barnes and Noble Pubit platform
Learn about Apples publishing platform
Find a new blogging platform
Create an open source project, rst2epub2, that converts reStructuredText to epub
Create an open source project, epub-css-starter-kit, that provides sample css to get started with ebook creation
Learn more about SEO
Mess around with Google adword and MS AdCenter
Buy a few domain names
Get involved with the e(book)production community

In the end though, I think I have the seeds of the Python book I would have wanted when I was learning Python.

Status of Book

The book is currently available in draft form. With the help of my wonderful technical editor, JJ, I am in the process of reviewing and rewriting portions of the book. If you, or a co-worker, or friend, or family member are interested in the book, you can certainly purchase a copy. All purchases will get the updated final version when it is released.

From all my adventures delving into ebook creation, it appears that there probably four things that determine success as an other (where success is defined as moving some non-zero sized amount of books):

People like or respect you
You have positive reviews on Amazon
You have more than one book available
Luck

If you are inclined to help me with any of the above, I in turn would be inclined to get you a copy of the book. As I'm just starting my new blog (as well as migrating content over from the old blog), I need all the SEO juice, links, and help I can get.

What's next?

What is next? I plan on sharing my experiences. I have already started sharing some code on GitHub. I've made some attempts at marketing, and hope to make more. I figure that I should be open and share my successes and failures, so that others can learn. They should appear on this blog, intermingled with other somewhat related content. If you have requests or ideas, especially if you are working with ebooks, I would love to hear them.

In the meantime, I have become somewhat obsessed with ebook creation. I have two other books I am currently working on that are related more directly to the ebook production process. I have also spent that last 8 Fridays teaching a group of ten 3rd-6th graders about ebook creation. Finally, I have submitted a PyCon talk about ebook creation using Python. So all of this is potential fodder for the blog.

If you have and suggestions, questions, or comments please feel free to contact me. Cheers!

"Creating Category Feeds in Octopress"

2011-11-09T06:31:00-07:00

The static blog generator, Octopress, does not generate per category RSS feeds by default. I thought I would need a plugin to do this but it is quite easy and a plugin is overkill. All that you need to do is create a feed file with the appropriate filtering to limit the category.

To create a per category feed, copy the default feed source/atom.xml file to say source/emacs.xml. I updated the <title> value and the for post loop to only use the emacs categories. Here is my file:

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title>{{ site.title | xml_escape }} - Emacs</title>
  <link href="{{ site.url }}/atom.xml" rel="self"/>
  <link href="{{ site.url }}/"/>
  <updated>{{ site.time | date_to_xmlschema }}</updated>
  <id>{{ site.url }}/</id>
  <author>
    <name>{{ site.author | xml_escape }}</name>
    {% if site.email %}
      <email>{{ site.email | xml_escape }}</email>
    {% endif %}
  </author>
  <generator uri="http://octopress.org/">Octopress</generator>

  {% for post in site.categories.emacs limit: 20 %}
  <entry>
    <title>{{ post.title | xml_escape }}</title>
    <link href="{{ site.url }}{{ post.url }}"/>
    <updated>{{ post.date | date_to_xmlschema }}</updated>
    <id>{{ site.url }}{{ post.id }}</id>
    <content type="html">{{ post.content | expand_urls: site.url | xml_escape }}</content>
  </entry>
  {% endfor %}
</feed>

"Removing Mobile Chrome in Octopress"

2011-11-07T22:40:00-07:00

On mobile devices, the address bar takes up too much space. Here is how I made it disappear on this blog. This blog is an Octopress blog so I put it in octopress/source/_includes/custom/head.html, but the code html and javascript should work on any webpage.

<meta name="apple-mobile-web-app-capable" content="yes" />
<meta names="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<script>
window.addEventListener("load",function() {
    setTimeout(function(){
        // Hide the address bar!
        window.scrollTo(0, 1);
    }, 0);
});
</script>

Just throw that in your HTML head and you should be good. Props to SO and David Walsh.

"E-books: It's Just Text"

2011-10-19T21:19:00-06:00

I'm volunteering at my kid's Elementary School to teach eight 45 minute classes. There are various suggested pre-canned courses that one can offer to teach. But I choose to teach a course of my own creation, How to Create E-books. So, I'm currently in the middle of teaching ten 3rd, 4th and 5th graders HTML and CSS. I've also taught them how to count in hex so they can use colors like the cool kids, but that is another post.

We aren't using fancy tools in this class like desktop publishing systems or even word processors. We are using perhaps one of the poorest text editors around, notepad.

The Dirty Secret

Most e-books are just HTML. While this is probably obvious to technical people, most people don't have a clue and don't care. I find it interesting that we live in a world where 3 year olds know how to navigate a smart phone, yet most elementary kids have a very rudimentary understanding of computers and how to use them. Still, they can master the basic concepts of XML (or HTML) and CSS relatively quickly. Most if not all of my problems have been when they have to do tasks that I thought were would be straightforward. Things like save files, open them in a text editor, or understand why they have to have an extension when the file explorer hides it from them. Ok, maybe not completely straightforward, but enough to make me wonder if most computer labs are not just excuses for letting the teacher recharge while kids play online games.

Under the guise of teaching writing, I'm subverting their little minds to learn something deeper that I doubt they will have a chance to learn in any traditional class during their K-12 years. Once you understand HTML, you can create web pages, e-books, understand how word processors might work, and have a notion of encapsulation and the engineering principles of building on top of existing technology. The black box of the web should be a little more clear to them.

The Basics

The basic rules of XML are easy for a 3rd grader to understand.

There are start tags and end tags. A start tag is just enclosed in brackets. Here's is a start tag for a paragraph, <p>. End tags have a / before the tag name. Here's the corresponding end tag for a paragraph, </p>.
If you open a tag, close it. This is good:

A paragraph with emphasized text.

This is bad, because the <em> tag is not closed:

<p>A paragraph with <em>emphasized text.</p>

Sometimes start tags and end tags are combined. For example an image tag normally looks like this :

The / before the > indicates that this image tag is closed.

Tags can have attributes. The src='cow.jpg' in the image tag above is an attribute. src is the name of the attribute, and in this case indicates that the value will be where the image where be found. The value of the attribute is always found following an = and quotes. Here's another example might be:

Treat opening paragraphs differently.

The above attribute has the name of class and the value of opening. Class attributes are usually combined with css to style certain elements differently from those elements lacking the attribute.

And that's about all there is. 3rd graders understand this in five minutes after you give them some examples and ask them some questions that make them think.

Digging In

Epub files are just zip files, so you can open them and peek around (if they lack DRM).

I examined a recent technical e-book from a well known publisher and it only contained 20 html tags:

a
body
code
dd
div
dl
dt
em
h1
h3
head
html
li
link
meta
p
span
strong
title
ul

Using a utility like http://www.w3schools.com one can learn all of these tags pretty quickly. Most ebook readers support basic HTML, but don't try anything too fancy if you want to work across different platforms and devices. And that is probably sufficient for most texts that are meant for reading.

A challenge

If you are writing an e-book (or a website/blog) and have not ever created content by hand, give it a try. You don't have to use notepad. You will realize that it is not black magic. And if you happen to view generated output from a fancy WYSIWYG editor, you will see all the cruft that they create. And you will learn why people make a living removing and cleaning up all of the cruft generated by such tools.

"Utah Python Oct 2011"

2011-10-13T00:26:00-06:00

Utah Python will be holding the October meeting on Thursday the 13th. We will have two presentations this month. The first will cover profiling of Python applications. The second will cover regular expressions. For someone who threw out the regex hammer from my toolbag, I'm interested in perhaps replacing it with an ice pick.

"KindleGen Fun"

2011-10-10T22:31:00-06:00

Can someone please tell me why kindlegen can't handle <pre>?

I have some html:

<pre class="literal-block">
$ python hello.py
</pre>

And kindlegen kindly gives me this?

<blockquote>
  <div align="left">
    <tt>
      <br/>$ python hello.py<br/>
    </blockquote>
  </tt>
</div>

Calibre on the other hand seems a little more pleasant::

<blockquote height="1em" width="0pt">
  <tt>
    <br></br>$ python hello.py<br></br>
  </tt>
</blockquote>

I thought well formed XML was a solved problem. The internet abounds with people claiming that one should only use kindlegen. The hacker in me sees a new rst translater, the realist in me cringes.

"A New Blog"

2011-10-03T22:04:00-06:00

tl;dr

Learn markdown, use octopress

Pending shutdown

A while ago, I was informed that my old blogging service would shutting down at the end of 2011. They served me well over the years, but this seemed like a good reason to move on. The first thing I did was write a crawler to backup my old website. The admin section of my blog had a button to download an archived tarball for each month, but I didn't feel like doing the manual labor clicking on that button some 60 odd times.

I chose to automate the downloading. Knowing that I wanted to do a new fangled static website, instead of getting the full high resolution version of my blog, I lossily encodend the bits into into reStructuredText. I figured that would be sufficient if I wanted to import the posts, since my next blog would probably based on Python and hence support rst.

Though, when I get around to importing the old posts, I'm not quite sure what to do about comments.

Why reStructuredText?

rst is the standard lightweight markup language for Python programmers. So I tend to favor it when possible. In fact I've gone out of my way to use it for writing presentations and even my book. Sadly, it's not the only markup I use. Being an emacs user, I also use org-mode for note taking. Sadly those two are not the only markups I use, being a confluence user at work, I use their wiki markup as well, but I'm digressing.

I set out on the task of finding my blogging system that preferably was written in Python (if it needed hacking) and supported rst. Then I realized {"I was not up for the task of writing my own blogging system. I wanted something that worked out of the box."} The best one I found didn't support reStructuredText. In fact it wasn't even written in Python. It supported markdown, which is should be okay because it seems like all of these markups support the same 70% of use cases. The annoying thing is to remember the remaining 30%.

I don't think I'm completely satisfied with any of the lightweight markups I've come across. But given the tools that I use, it has made sense to learn the markups associated with said tools.

A typographic aside

I've been reading The Elements of Typography and have come to the realization that current lightweight markup languages are not sufficient in their support for typographic markup. For example, there is no way to designate dropcap support. Hacks like smartypants sit on top to provide functionality like curly quotes. Not to mention that typographic support for the web is years behind the renaissance standards. Ligatures? Maybe with some ugly javascript hack we'll start seeing those in a five years, though they have existed in print for centuries. (Did you spot the missing ligature in that sentence?) Sadly there is not enough time in the day to do much about this.

So the downside of reading that book is that I now take a hyper critical eye for some typographical nuances. These were things that I didn't even know about before and didn't have to spend brain cycles on them.

Enter Octopress

I followed the various shoutouts on Hacker News and the blogosphere regarding static blogging packages. I've watched them in github and I've even perused the source of some of them to evaluate them.

After attempting to get two different Python blogging systems up, and realizing that it required more hacking than I was motivated to do, I re-evaluated.{" Currently Octopress seems to be the best static blog system for me."} Other than the initial ruby install (it requires version 1.92, and apparently Gentoo devs aren't into that), it has been smooth sailing. I haven't had to touch ruby, only markdown and yaml.

It looks nice out of the box. It has lots of action on github. It supports code fragments, quotes, and pull quotes quite easily. It even looks decent on mobile devices. To enable remote deploy I had to update 2 lines of yaml.

I guess I've been in the mode of thinking I'll eventually need to hack on a given tool. Yet the pragmatist in me really appreciates that I can just use Octopress as a tool.

I don't think I'll be leaving Python soon. Octopress is a great tool (it could be written in Python or Erlang for all I know) that does it's job out of the box. I'm a little remiss that it doesn't automatically import clipart and write my copy for me, but I think I'll get by.

New lappy, Lenovo T61p (linux edition)

2008-06-26T00:00:00-06:00

My trusty r52 switches between a state of 100-200 free megs of memory and no memory on the 40 gig hard drive. Rather than investing in an expensive IBM harddrive (I've heard the BIOS on my machine won't take normal OEM drives), I got a new computer.

Since I'm a freetard, I got the 15.4 inch t61p linux edition. I immediately installed Gentoo on it, and was disappointed that Linux/thinkpad support has gotten worse. Getting the wireless to work was a pain (I thought Intel was supposed to be open source friendly these days). Probably my biggest gripe is that the volume buttons are now controlled via software rather than in the hardware (may seem like a minor gripe, but it seems like I use those buttons a lot, especially if I'm listening to pandora or archive.org). Perhaps my problem was that I was also running KDE4 (I spend most my time in Konsole/emacs/firefox so I thought I could get away with it (actually Konsole in KDE4 is awesome, the rest felt very beta-y)). The whole experience was quite frustrating.

Oh, and the widescreen model doesn't fit my timbuk2 laptop bag. More frustration...leaving me realizing that I probably prefer everything about my old laptop (except for the aging battery, full harddrive, single core, and missing right alt key).

So after some 7 years with gentoo, I installed Kubuntu (3.5) on it. Right now I'm pretty happy with it. Most of my gripes are now gone. Though I'm fearing what will happen in 4 months when Intrepid comes out. I'll probably be wishing I was back in gentoo land.

One cool feature is that some common applications, when you try to run them the system, doesn't only tell you the app isn't installed, it tells you which app to install. Very nice.

Haven't got vmware running yet. It keeps crashing. Seemed to just work on Gentoo...

So if I post some Ubuntu findings, you'll know the reasoning.

Why doesn't pushd popd work in my Ubuntu shell script?

2008-06-26T00:00:00-06:00

I'm in the process of moving over to my new laptop. I run into an issue where a shell script (that worked under gentoo) now no longer works. Here's the error

foo.sh: 18: pushd: not found

Since I'm running this as

sh foo.sh

I turn on debugging with

sh -x foo.sh

Which doesn't help because the result is the same. Weird! I can run them from the command line.

Some searching led to my answer. Pushd and popd, are actual binaries, but are builtin to the shell. The default shell in ubuntu, dash doesn't have these commands. So I need to either invoke my shell as

bash foo.sh

or put

1	#!/bin/bash

as the first line and chmod it, and execute it directly. Now as to why pushd/popd work from the command line, ~~still haven't figured that out~~ because shell is bash.

pyExcelerator (xlwt) cheatsheet (create native Excel from pure python)

2008-04-26T00:00:00-06:00

If you are looking to generate/create/write (but not read) excel spreadsheets from native python (read on linux or macos or even Windows!), then xlwt (a fork of pyExcelerator) is your friend. This library, (and the ancestor) is somewhat unknown, but quite good. The only real problem with using it is that documentation (and the bug I mention below), (though this blog post is pretty good) is somewhat spotty. But you can usually get around dearth of docs by reading the code that reveals that the internal model for .xls documents is quite good, and that is supports most features (sans graphing). The original MIA author did quite a good job.

I've made a cheatsheet(pdf) for these tools. Inkscape still has some issues exporting this but here's the source for it (and the formattings bad, but I thought I'd just get it out there). It's an executable cheatsheet, so if you run it (the .py not the .pdf), it will create a spreadsheet. Hopefully this is useful to some folks. I've also tried to embed docs into the code that will serve as useful hints.

Also, I'm using a factory to reuse styles and fonts, as it seems that if you define too many of them, Excel has problems and crashes (note that OOo won't manifest them and happily plugs along on the non-conforming files). Please send any feedback or comments on how this could be more useful.

Also note, this is the .xls format, not the newer xml/ISO/.xlsx stuff

#!/bin/env python
"""Executable cheatsheet illustrating use of pyExcelerator and its
fork, xlwt

I recommend using xlwt which is a somewhat unknown fork of
pyExcelerator.  There are examples shipped with both projects, use
them if necessary, but the source is usually your best friend.  The
libraries are quite capable but don't support charting.  xlwt also has
a mailing list that is active here:
http://groups.google.com.au/group/python-excel

Another good link is here:
http://ntalikeris.blogspot.com/2007/10/create-excel-file-with-python-my-sort.html

This illustrates common usage for .xls generation, but also uses
factories to limit object creation of styles (which can crash Excel).
It's meant to show example, but for more details, I recommend the
sources I mention above.

Please send comments/suggestions my way

author: matthewharrison@gmail.com
"""

#import pyExcelerator as pycel
import xlwt as pycel

# Excel has issues when creating too many styles/fonts, hence use
# a factory to reuse instances (see FAQ#13 http://poi.apache.org/faq.html )
STYLE_FACTORY = {}
FONT_FACTORY = {}

def create_spreadsheet():
    # Create a workbook
    wb = pycel.Workbook()

    # Add a sheet
    ws = wb.add_sheet("Example Sheet")

    # Tweak printer settings
    # following makes a landscape layout on Letter paper
    # the width of the columns
    ws.fit_num_pages = 1
    ws.fit_height_to_pages = 0
    ws.fit_width_to_pages = 1
    # Set to Letter paper
    # See BiffRecords.SetupPageRecord for paper types/orientation
    ws.paper_size_code = 1
    # Set to landscape
    ws.portrait = 0

    # Write some stuff using our helper function

    # Formatting - hint, look at Format code in OOo
    #              format cells... Numbers tab
    # Write a percent
    write(ws, 0, 0, .495, {"format":"0%"})
    # Write a percent with negatives red
    write(ws, 1, 0, -.495, {"format":"0%;[RED]-0%"})
    # Dollar amounts
    write(ws, 2, 0, 10.99, {"format":'$#,##0'})

    # Font
    # Size
    write(ws, 0, 1, "Size 160(8pt)", {"font": (("height", 160),)})
    write(ws, 1, 1, "Size 200(10pt)", {"font": (("height", 200),)})
    # Bold
    write(ws, 2, 1, "Bold text", {"font": (("bold", True),)})

    # Background color
    # See http://groups.google.com.au/group/python-excel/attach/93621400bdddf464/palette_trial.xls?part=2
    # for colour indices
    YELLOW = 5
    write(ws, 3, 1, "Yellow (5) Background",
          {"background": (("pattern", pycel.Pattern.SOLID_PATTERN),
                          ("pattern_fore_colour", YELLOW) )})

    # Border
    write(ws, 0, 2, "Border",
          {"border": (("bottom",pycel.Formatting.Borders.THIN),
                      ("bottom_colour", YELLOW))})


    # Wrapping
    write(ws, 0, 3, "A bunch of long text to wrap",
          {"alignment":(("wrap", pycel.Alignment.WRAP_AT_RIGHT),)})

    # Set column width
    # (see pycel.BIFFRecords.ColInfoRecord for details, width in
    # 1/256th of zero character)
    write(ws, 0, 4, "A bunch of longer text not wrapped")
    ws.col(4).width = len("A bunch of longer text not wrapped")*256

    # Freeze/split headers when scrolling
    write(ws, 0, 5, "Header")
    ws.panes_frozen = True
    ws.horz_split_pos = 1
    for row in range(1, 200):
        write(ws, row, 5, row)

    # Save the workbook
    wb.save("out.xls")

def write(ws, row, col, data, style=None):
    """
    Write data to row, col of worksheet (ws) using the style
    information.

    Again, I'm wrapping this because you'll have to do it if you
    create large amounts of formatted entries in your spreadsheet
    (else Excel, but probably not OOo will crash).
    """
    if style:
        s = get_style(style)
        ws.write(row, col, data, s)
    else:
        ws.write(row, col, data)

def get_style(style):
    """
    Style is a dict maping key to values.
    Valid keys are: background, format, alignment, border

    The values for keys are lists of tuples containing (attribute,
    value) pairs to set on model instances...
    """
    print "KEY", style
    style_key = tuple(style.items())
    s = STYLE_FACTORY.get(style_key, None)
    if s is None:
        s = pycel.XFStyle()
        for key, values in style.items():
            if key == "background":
                p = pycel.Pattern()
                for attr, value in values:
                    p.__setattr__(attr, value)
                s.pattern = p
            elif key == "format":
                s.num_format_str = values
            elif key == "alignment":
                a = pycel.Alignment()
                for attr, value in values:
                    a.__setattr__(attr, value)
                s.alignment = a
            elif key == "border":
                b = pycel.Formatting.Borders()
                for attr, value in values:
                    b.__setattr__(attr, value)
                s.borders = b
            elif key == "font":
                f = get_font(values)
                s.font = f
        STYLE_FACTORY[style_key] = s
    return s

def get_font(values):
    """
    'height' 10pt = 200, 8pt = 160
    """
    font_key = values
    f = FONT_FACTORY.get(font_key, None)
    if f is None:
        f = pycel.Font()
        for attr, value in values:
            f.__setattr__(attr, value)
        FONT_FACTORY[font_key] = f
    return f

if __name__ == "__main__":
    create_spreadsheet()

Lasers/Wiimotes/Python presentation at Utah Code Camp

2008-04-25T00:00:00-06:00

For those in the Utah vicinity, and who want to see a cool presentation about python, wiimotes and lasers (or who really like MS products, since most sessions seem dedicated to that), my brother is presenting at Utah Code Camp. It should be a little cooler than the Pycon version, since some more stuff has been added.

Oh, and I guess there is one other python talk amidst the sea of MS technologies, Kevin Kubasik is talking about twill and lxml.

[Pycon 2008] Lasers, Webcams, Wiimotes and Python Video up

2008-04-25T00:00:00-06:00

My brother gave a talk about using python to detect lasers with webcams and also demo'd a prototype of a 3d game with wiimote headtracking. For those who missed the preso, there's now a video up.

GSOC Project for Branch Coverage reporting for Python

2008-04-02T00:00:00-06:00

As there appears to be some interest in branch coverage and the cool testing/metrics/quality tools that can be built upon it, I've proposed a GSOC project to start the ball rolling. (I assume it will start rolling either way, but allowing someone to devote more time to it is always nice).

I put a short writeup on the Python SOC wiki page. Here's a little more info. There's only so much one can do with line coverage, but that's the best python has at the moment because the one can only trace at a line level. If tracing were enabled to happen at a condition level, then branch (and path) coverage can be reported. For people who want to go hog wild and try to do full path coverage, they can go ahead. For those who want a little more than branch coverage, there is Structured Testing. (For people who don't want to go that far, perhaps you have a bunch of tests that take a while to run, if we have path data, it would be able to tell you which tests were redundant and you could cut your testing time).

Here's my thoughts on what the project might include:

Update sys.settrace to enable branch coverage (or add a new function)
Create library to implement branch coverage using above tool
Create command line app using said library to implement branch tracing (aggregating subsequent runs) and dump/pickle data
Create command line app to load and report on data
(Test above)
Bonus stuff
Calculate flow graphs
Calculate basis paths through graphs
Take branch reports/basis paths and determine missing paths/tests
Create visualization for flow graphs/extend Ned's work
Create visualization for basis paths (in pygame?) that links code to flow graphs and tests to executed paths
Redundant test finder
.... cool stuff you think of

I'm willing to mentor this. If you are interested please or have suggestions please let me know. Comments? Other ideas?

Undeleting photos on (gentoo) linux/windows/mac

2008-03-24T00:00:00-06:00

This post is preceded by a public service announcement to back up all important data, often. (So far it seems the best way to back up photos is to get them printed/developed).

Had a scare with a half gig compact flash card acting up on me, that contained photo documentary of my daughter's birthday. Enter Gentoo Wiki to the rescue. (Note that this site contains information that could be useful for other distributions as well, like say fedora or ubuntu....).

Here's a nice link explaining different steps to take to recover photos. A few notes, are that:

dosfsk

runs in place (seems obvious), but when I have a wacked out drive/partition, I really want to copy it (using dd or something) and work on the copy, not on the original.

Anyway, dosfsk claimed to fix/rename a bunch of files. Then I started trying to use

jpg-recover

I had issues with it (it's slow and it seemed to hang my computer). Plus I don't think just recovering jpg's is sufficient these days, when you have cameras that now double as camcorders.

So here's the most useful advice. Use PhotoRec which also recovers movies. It uses a (semi-confusing) wizard interface. The User guide walks one through the steps. The most confusing issue was selecting the partition table (what is an Intel partition?). The Canon (fat16?) format was recognized when I selected "None" as the partition table type, but not when I tried Intel partition. After that it ran (really quickly) and recovered most/all of the files.

The homepage of photorec says that it runs on windows and macs too. So if you find yourself in need check it out!

[Pycon 2008] Linky linky

2008-03-21T00:00:00-06:00

Here's my notes on Pycon this year. I'm going to throw out a collection of links that may or may not be useful to you. If you weren't able to make the conference, hopefully this list contains a link or two that are new or interesting to you.

elasticwulf slides)
ipython1 is a branch of ipython for running parallel jobs.
elasticwulf is a set of python tools for controlling beowulf clusters on ec2.
mpi4py is python bindings for mpi.
http://del.icio.us/tag/ec2+gotcha )
pyke is an expert system written in python. (Attempt at rewriting prolog in python?)
gdata-python-client provides tools for reading/writing google data from python (programatically update spreadsheets, etc)
Firefox EC2 Extension allows one to interact with ec2 instances from a browser.
s3organizer extension allows one to interact with s3 from a browser.
boto is a python interface to amazon's webservices, s3, ec2, etc.
paramiko for scripting ssh sessions with ec2 instances.
Normal accident theory if you call echo "">/dev/null on too many instances they will crash!?!?
Trestle is a nose plugin for writing doctest like tests for webapps.
Thrift is RPC for facebook (NIH). It has python bindings.
haufe.eggserver allows for local pypi repositories. Could be useful when building your projects that depend on wheels that are constantly moving.
Python metaweb/freebase client code allows for querying the metaweb online database (like wikipedia, but graph based).
Ella is a cms for django that announced their open sourcing? at the conference.
Use argparse for parsing command line scripts if you don't mind pulling in one more library and optparse isn't quite doing it for you.
Orbited can be used as a comet server (server push instead of client pull).
Wiimote head tracking/laser/webcam blog very cool demo code for wiimote head tracking in 3d enviroment and webcam/laser detection in python. (Disclaimer: this is my brother's blog, I've had a small hand in some of this)
Traits is an event system for python
Five dash is python based open source accounting software (think quickbooks).
Geodjango extension to django to help with geospatial data. Jacob said this will go into the trunk soon.
Openlayers js library to do googlemaps type interfaces.
snpedia allows you to upload your dna and tells you your traits (either good or bad) has a social/wiki aspect to it.
geopy allows one to find coordinates of addresses, cities, etc.
pvote lightweight (<500 loc) voting engine, that should be easily certified as "secure".
TGPPL is a new license that morphs from GPL to BSD overtime. Thought to follow same ideas as IP/Patent protection.
pyprocessing is an attempt to get around the GIL by making a library that uses processes that has an interface very similar to the threading interface.
pprocess is another GIL hater.

[Pycon 2008] Thoughts and Observations

2008-03-17T00:00:00-06:00

Here's my thoughts regarding Pycon 2008.

Benefits of conference attending

Face time with virtual friends - Some people/IT shops invest in established technology provided by big vendors. I've been somewhat disenchanted by the products (and underlying lockin/intentions) of many vendors. Yet open source (for the most part) has provided wonderful cutting edge solutions and tools. As you invest in these tools such as these, you enter this "onion of open source" (by onion I mean consider a sliced onion, where the layers correspond to moving deeper within a project, ie: lurker, downloader, user, mailing lister, bug reporter, patcher, ircer, committer, ...).

Just as you invest (money) in big companies for their tools, with open source, you invest time. But the paybacks for developers can be great. Rather than interacting with a huge faceless entity, oftentimes you can come mingle with the primary developers directly.

It's very nice to be able to say "Thanks for your excellent software, and your help" in person.

Get up to date - I don't have time to read blogs all day long, so sometimes it's nice to get a firehouse of cool stuff that people are doing, that may or may not be directly related to your work or interests. A conference such as Pycon or OSCON can provide this.
Network - I don't consider myself to be a good networker, but if you are interested in something (like say python), it's good to be in an environment surrounded by people who probably share some subsets of interests with you.
Get a job - Perhaps this is related to the previous (but since I'm not looking for a job, I'll separate it). Vendors come to hire, and they realize that people attending a conference are probably more "into" the subject of the conference than those who don't attend, so it's one way to get a "foot in the door". Though, the older I get the more I realize that good job opportunities will probably find me via my network, than less effective means (such as me spamming my resume).

Food

Food quality went down this year. I guess the box lunches scale better, but the taste goes down. Also, previous years have had talks during lunch. This year lunch was the hallroom track. At least put vendor/lightning talks during that time.

Session quality

Disclaimer: I gave a session and convinced my brother (who was a python newbie at the time) to submit a session (which he gave).

Having attended a few open source conferences (OSCON, PyCon, Foss.in, Mysqluc, OSBC, LinuxWorld, Mashupcamp, UTOSC) and user group meetings, I realize that sessions are a mixed bag. You've got vendors, who want to tell their story to any eyeballs they can. There's the community figureheads who want to get their point out. There's community members who think they've got something interesting to say. Then you consider the level of the talk: beginner, newbie, advanced, requires 3 PhDs to understand. There's also the nature of the talk itself, is it more breadth or depth oriented?

I can usually tell within the first five minutes whether the talk will be interesting or valuable. The value derived from a talk can come in different ways. Sometimes, a URL or two are the most important parts of the talk. Sometimes the whole talk is interesting. Other times the title sounds interesting, but the content is unrelated or presented at a level below or above what was indicated.

Regarding Vendor/Sponsor talks. I'm comfortable with them. What I'd actually prefer would be a lightning session dedicated to them. Ask the vendors to describe the work they are doing, and where they are located. Don't tell me how nice the office is, or the view, or the size of your monitors. If I'm interested in working with you (it will probably be because your work is interesting), and I'll come and find you and then you can try and sell me on the other benefits (monitor/office size etc). Or preferably if you know who I am and what benefits I can provide, you can contact me. (Perhaps the conference registration could allow job seekers to note their skills/blog/projects and if they are interested in being recruited).

Regarding keynotes. I think there are too many. Rather than two 20 minute talks and an hour talk, just give me one or the other. And then have another session or two, that would give some room for another 24 talks. (Which seem to have about the same hit/strike ration as the keynotes)

Regarding talk quality. That's a hard one. You want the good consistent speakers, but also don't want to exclude other's who might be good, just not known. Perhaps a preparation checklist for speakers might be nice. I try to practice at the local user group previously. Also, I like to state the level of the talk and what I'll be covering up front. That way if there are two talks you are split between (and strangely there seemed to be a few sessions where the tracks seemed to overlap quite a bit), and mine isn't what you really thought it would be you can skip out and try to make the other one.

Another thing that is useful (but is after the fact) is letting the speakers actually view they videos of themselves. Then they can see how they look to the audience. Personally I can say that it is a scary and humbling thing. But the only way to get better at speaking is to practice, and make note of what you do good and what you could do better.

Here's another hint: Don't read #pycon before you have to talk, its somewhat ruthless nature (towards speakers) can be quite intimidating ;)

Regarding talk level. If you want to grow the popularity of python (admittedly that is not the goal of everyone, some like to be exclusive with their tools), then you need to cater to all segments, beginner, intermediate and advanced. Perhaps some (good) beginner talks will recur year after year (or every other year) because they are timely, yet useful. Perhaps some advanced topics can span two sessions. For example, I've programmed in python since 2000, but have never ventured into the c guts. A session (or two) on how to do that might be nice (it could also help find new core contributers), and it could be useful year after year to a new crowd. I'm pretty sure an Intro to Twisted would be packed about every year....

Regarding talk content. I can and have spoken about many different python subjects. But when I apply to speak, I try to speak on something that I think is interesting or useful, but not necessarily what the audience thinks is useful. I think speakers like to speak on new or novel things (more academic). But there are talks at all levels that could be quite repetitive but still useful. Perhaps for a month or two previous to the call for speakers the organizers can publish what topics people want to see. Or perhaps there could be specific tracks (ie testing, web, scientific, gui, automation, etc).

Regarding open spaces. Some have suggested that pycon should be taken back to the community and become an unconference. I don't know that that scales to 1000+ people. (Or that it would attract that many people. It's a risky proposition to spend 3 days at a conference that could talk about anything). When I attended mashupcamp (a pure unconference) that was supposed to have the who's who (I was sitting in for one), I found 70% of the sessions extremely boring. It was probably worse than most conferences. But I guess with the Bofs/open space rooms available all day anyway, this could be/is done already. Note that I have found Bofs useful, I'm just not sure I want 100% Bofs.

** Vendor Swag

With ultimate (frisbee) leagues and conference attending, I have more than enough t-shirts. But I must say that the conference shirt and the Leapfrog Online shirts were quite funny. Good job.

It's always nice to get something the kids might like. But please, if your swag has the following warning:

Sonic Rocks are powerful magnets. Always keep them at a safe distance (at least 20 inches) from magnetic storage media ... such ... as ... hardisks ... and laptops.

Please tell me not to put it in a bag right next to my laptop. (There were quite a few laptop/presenter glitches this year .... possibly .... related???)

** Laptop/software thoughts

Apple appears to be the product of choice with about 2/3rds of the market here. The remaining third again split amoung linux/windows. Saw about 10 eee's, 2 macbook airs.

Most presentations were done in keynote or s5. S5's lack of image scaling is bothering me. Me thinks that clutter or bruce need needs an rst converter for slideshows....

Students (that I talked to) seem to be almost 90% on linux. Not sure if it is the geek factor, lack of windows fee (laptop already comes with windows so....), or they are just forward thinking/wishful.

** Attendance

Last year had 40% growth to 600. This year saw 1000+ in attendance. I'm not sure the venue will scale to next year. The halls were pretty crowded.

Saw DHH briefly in attendance on Saturday. Weird.

** Conclusion

No conference is perfect, but I had a great time this year. I highly recommend PyCon for developers as well as companies that are using python (not to pitch us, but to interact with the community and to form a sybiotic relationship instead of leaching one).

Ask Matt: Java, PHP or Python for beginning web development?

2008-03-06T00:00:00-07:00

I recently received the email below. At the risk of starting a flame war, I'm posting my response here.

Hi Matthew,

Sorry to bug you, but I was reading your blog and was hoping you could help me with some advice. I am new to programming and can't quiet make up my mind on which programming language to start with? I started learning php, but have also looked at a bit of java.

I am interested in building web based applications that and am wondering whether I should focus on php, java or python? I have recently read a fair bit about python and some exciting things happening using python. Python seems like an easy , fast and powerful language to use.

Do you have any advice or thoughts on the pros and cons of these 3 languages and what might be the best to start with?

Thanks for your help- and the great blog!!

My response:

Hey Sean-

PHP, Java and Python are all good languages and each has it's strengths and weaknesses.

For web development, php is going to be the easiest to start out with since it was designed for creating websites. However that easiness is going to be lost once you start trying to do more complicated things and inevitably your code will probably turn out quite messy. Lack of support for classes and name spaces for older versions of php tend to aid people in programming sloppily. On the other hand there are a ton of great applications written in php (drupal, mediawiki, sugarcrm, etc), so it is possible to use. If for example your website was based on drupal, you could get a lot of functionality out of the box, and only have to write a small amount of code for missing functionality.

Java has a few things going for it, a big company with marketing dollars and great tools. However the reason it needs good tools/IDEs is because the there is so much syntax that you really want the computer to write all that code overhead for you rather than your fingers. If you are going down the java route make sure to use an IDE like eclipse since it will make your life much easier. I've done a bit of struts programming in my day, and actually don't miss it so much. There's a lot of overhead involved with configuration files that you don't find in php, but that tends to help larger teams coordinate their efforts better. Since moving away from java, I'm not sure what the MVC stack de jour consists of, likely it will include spring and hibernate, as well as other packages.

I'm sure you realized that reading my blog I'm going to favor python. Python is a multiparadigmatic language, so one can program in an object oriented, functional or imperative style. The advantage of this is that there are cases where objects are overkill and get in the way. Sometimes functional constructs just make sense for dealing with some data structures, while an imperative style like C is often the most straightforward solution. Python lets you mix and match that, so with that flexibility also comes power and the ability to program in a way that suites your style. Java is strictly object oriented and PHP until recent versions was imperative with poor support for objects.

Setting up an web development environment is relatively straightforward in python. Both of the web development frameworks I would recommend in python, Django and Pylons, come with built in servers so you can get up and running pretty easy. In java land, struts setup to start a project was a major hassle, but new frameworks might have simplified this. In the php world, one needs to install apache and php, which may or may not be so easy (xampp can help here).

For python development I use emacs, but if you are an IDE type person there are plugins for eclipse, or IDE's such as wing or komodo that have pretty good python support.

I've known quite a few php/java heads who upon learning python haven't looked back. I don't know of any who have gone the other way. Though the lack of static typing/IDE support has proved to be an annoyance to some java developers who have grown accustomed to the niceties of eclipse (code completion, refacoring, error marking).

As far as the Django or Pylons question, Django is probably the best bet for you if you are learning to develop web apps. It has great documentation (plus a good book, that has a free online version), and a great community. It's nearly ideal for greenfield applications, and the admin app is really cool for adding/editing your web models. Django gives you everything you need out of the box.

Pylons on the other hand let's one pick and choose which components to use, rather than forcing one on you. Though the recommended defaults are pretty good. I'm currently using Pylons at work because I have to tie into an existing database, which SQLAlchemy supports wonderfully (and the django ORM just plain wouldn't work with it). So while Pylons gives one the power of choice, that also has an effect on documentation and other factors that make Pylons not quite as integrated overall as Django.

Hope that this (biased) answer helps to bring one more python programmer into the world.

Cheers,

matt

Fun with YUI, IE and JavaScript

2008-02-29T00:00:00-07:00

I still develop JS in basically the same way I did back in 2001. Back then IE had more of a stranglehold on the browser market, but it was a horrible JS development environment. So one would use Mozilla with the Venkman debugger to make things a little easier, then cross their fingers and hope it worked on IE. Where it didn't work, you'd put in a few if (document.all) { and some alerts until you figured out what is wrong. Today we have AJAX libraries so everything is cross platform.... yeah right. So for the sake of others who might run into these issues, I'm populating the indices of search engines.

`Select.add` issues

Like I said, I still develop on Mozilla (firefox) and use the debugger there (now firebug instead of venkman). Since I'm a freetard and run gentoo, I'm running IEs4Linux to occasionally check that my stuff works under IE6. My JS was throwing a "Error 80020005" error, and the search engines weren't returning anything relevant. But the problematic line was where I was using

select.add

as illustrated here in quirksmode.org (note that the quirksmode code doesn't appear to be cross platform). Turns out that firefox (and konqueror) are happy when the index is null, whereas IE wants an index number. I finally figured this out on my own, but later found that the example here also illustrates it. My code (since I'm using YUI's builtins for user agent checking) looks like this now and is happy:

if (YAHOO.env.ua.gecko > 0) {
  select.add(option, null);
}
else {
  select.add(option, select.length);
}

Using JS to extract option values

This one threw me for a while too. If you set options like this (and as illustrated in the example previously linked to) in html, you'll have issues in IE:

<select id="mySelect">
  <option>Apple</option>
  <option>Pear</option>
  <option>Banana</option>
  <option>Orange</option>
</select>

Make sure you put the value attribute in, or form.formname.selectname.value won't have a value in IE (but it will in firefox/konky).

<select id="mySelect">
  <option value="Apple">Apple<option>
  <option value="Pear">Pear</option>
  <option value="Banana">Banana</option>
  <option value="Orange">Orange</option>
</select>

`YAHOO.util.Connect.asyncRequest` caching on IE

Another issue I ran into was IE appearing to make ajax calls, but not actually hitting the server. (When it really just hit the server the first time, and used the cache for subsequent calls). Again, this was different behavior than FF/Konky. The secret was to add cache:false to the callback hash (required in the asyncRequest function).

this.callback = {
  success:this.handleSuccess,
  failure:this.handleFailure,
  cache:false
  }

I'll end now, I have a bunch more js ranting, perhaps another post. Sometimes I just wish JS was more like python and less like php/perl..... (yes the whitespace wouldn't really work in the xml/html embedded environment... but I digress).

The good and bad of Asus EEE

2008-01-08T00:00:00-07:00

Like many other nerds, my wife got an Asus EEE for Christmas. (ok she isn't a nerd but her husband is). (Ok, maybe she is a nerd since I've been forcing her to using Linux for so long that she doesn't mind too much anymore).

The machine is solidly constructed and it just works. Everyone comments on how they first thought it was just a toy. It even came in handy over the holidays as a Balderdash game (we didn't have the actual game but there was an online version played on a mailing list that we used. We just passed the machine around, instead of the cards)

What to say about the keyboard? My daughter in kindergarten loves it. (She also loves playing frozen bubble, tux racer, tux paint and composing notes to her friends in OOo). My sister in law was able to type 70 WPM at 97% accuracy in the first two minutes of using it. (I wouldn't say she has small hands, but she normally goes 90 WPM).

Screen size? A 10 inch screen would be fine, but for an ultraportable machine, I'm not complaining. Plus there's vga out if you are using it for long periods of time.

My biggest gripe? Actually it's the operating system. No, I don't mind Linux. The simplified interface is fine. And everything just works, wireless, suspend, etc. Ok, it's not really the operating system per se, but the repositories behind it. Basically there are none, and because you are running a fork of a fork (Xandros is a fork of Debian, and the EEE distro is a fork of that), compatibility is a crapshot. And because it's not my machine (again this is my wives), I'm not going to go off installing stuff on it. My wife actually likes the gimp! And the EEE repos don't have this very common package...

It'd also be nice to have a more recent version of OOo (it's at 2.3.2 and EEE has 2.0??!!). I'd like to maybe put some of the dumb games I've done, or some of my brother's cooler ones on it, but that would probably breaking something.... It actually comes with python 2.4 and 2.5, so pure python is ok for development, but if you want to wrap anything that isn't included (or need wrappers) you are up a creek.

I'm sad that Ubuntu didn't jump on this. They've got the repositories and the community. And this would be great marketing to others. You could say, "hey, you can run this OS on your normal machine too". I realize there is eeebuntu, but I'm talking about out of the box, not having to worry about installing anything. I think ASUS and Canonical would both be a bit better off with their great products if they had partnered on this disruptive tech.

Memory profiling of python apps

2007-11-29T00:00:00-07:00

A much as some gnomers don't like python ;), there's some good stuff coming from those that do. Andy Wingo writes of his work to reduce the memory footprint of the Flumotion Media Server. It is linux specific (and somewhat gnome/gtk specific), but if you're looking to profile memory usage (and your code somewhat runs on linux), this will be useful.

MonetDB and SQLAlchemy

2007-11-19T00:00:00-07:00

I've spent the past days starting a MonetDB backend for SQLAlchemy. My code is currently attached to ticket 874.

Here are my thoughts after spending a bit hacking out the backend (which is really slightly usable if you have nightly of Monetdb):

I was thinking writing the backend would require me knowing indepths the guts of how monetdb handled sql. It turned out to be pretty easy. I just used kdiff3 to look at the backends of firebird and sqlite (since they were short). The basic idea of a backend is to wrap dbtypes to SQL statements (when used in columns) and convert types back to native python types when pulled out of queries. Then there is a Dialect object that knows how to reflect ones database. Most of my time was really spent getting that working. The only other thing that (so far) has proved annoying is auto increment columns, I'm using a combination of 4 booleans to determine that (including one called autoincrement....).
Though SA has a big wrapper around unittest that gave me a few minutes pause, I finally figured it out. Writing unittests was a great way to get some basic functionality.
It'd be nice if SA included a SQL99 or basic compliance testsuite for all the backends. All the tests appear to be silos of tests that may or may not overlap. I pulled/modified tests from sqlite and mysql for monetdb and there wasn't much/any overlap in those tests...
On that note I started writing doctests against the documentation for SA. This revealed a bug in monetdb, that hopefully should get fixed soon.
So, while the code is still in a nascent state, testing (both unit and doc) have allowed me to progress quite far and quickly.
The Monetdb people have been a pleasure to work with. They have been swift to answer questions and fix bugs (3 so far...).
One final note. The world of opensource is small. I previously had the pleasure of working with one of the monetdb dev's on a non-python, non-db related opensource project. Cool how you can run into great developers again.

So if you are in the market for a column-oriented db and using python (and feeling brave and adventurous) give the new backend a try.

Intro to python testing talk (unittest, doctest, coverage)

2007-11-07T00:00:00-07:00

I'm giving an introductory talk on testing tomorrow for Utah Python. I'm planning to cover unittest, doctest, coverage and interpreting coverage results. At the end will be a hands on session for applying what we've learned. We'll write a simple program and then try and achieve good test coverage using unittest, doctest and figleaf. If you don't have a laptop, you can pair with someone who does.

If you're in the Salt Lake area, I'd invite you to come out. We don't bite too hard. Plus there's usually food and beverages.

Mono, Miguel and Matt Asay

2007-11-02T00:00:00-06:00

As long as biased people are commenting on Matt's pining for a more innovative Miguel, I thought I'd throw in my two cents. I'm certainly biased* and have my opinions as well.

So yes, Matt made a technical faux paus in pleading with Miguel to develop more applications like Dashboard instead of wasting time on copying C# and MS. (Dashboard is in fact a C# app).

Here are a few thoughts of mine based on my industry experience and working directly with many open source shops:

Thoughts on MS/C#:

C# is a slight improvement on Java. I would call it MS's attempt to attract Java devs while completely controlling the platform (unlike J#).
C#, though open, is a MS technology. Though Mr. Hilf is changing some things up at Redmond, I'm still weary of them.
Adoption of C# is from MS shops (obviously)
MS shops are very unlikely to adopt or approve non-MS products and "Open Source" is the last thing they would adopt due to all the FUD that MS has been spreading about them.
Given the above reasons, I don't think Mono is going to help Linux adoption. It further entrenches MS's monopoly.
MS is smart to focus on adding other languages to the CLR (ironpython, ironruby....)
One tangible benefit of C# and the .Net platform is that it might have been an impetus towards the opening of the Java platform.

Thoughts on Gnome:

I agree with Matt here. Miguel has spent the last X years copying MS IP rather than innovating on the desktop.
The two biggest Gnome based distros (Redhat and Canonical) don't do Mono development
The biggest Mono adopters are .... Novell! Surprise surprise! Sort of ironic since SuSE was traditionally the biggest KDE desktop....
Mono is no silver bullet for taking over the desktop.
Is it really that hard to program with pygtk? (Ala redhat and canonical)

Thoughts on Matt:

No Matt isn't technical. But he knows the Open Source industry inside out. Moreso than anyone else I know. I don't agree with everything he says, but I think he has a point here.
Just like Miguel, Matt has changed over the years. Our first OS discussion was how to get around the GPL while he was at Lineo. Later I actually made fun of him for carrying around a .NET book (previous or directly after Ximian acquisition). He's now pretty pro-GPL...
Matt blogs. A LOT. Your rss reader will be busier than watching SteepandCheap.
Matt runs OSX... ;)

Thoughts on Miguel:

Re-implementing Silverlight in a month or whatever is quite an engineering feat. Congrats!
You're welcome to scratch whatever itches you

Full disclosure

*Matt is a friend of mine, we're having dinner tonight. I've actually used a mono program, Hugin.

Reinteract - nifty (editable) python shell

2007-11-02T00:00:00-06:00

Just watched the screencast here about Owen Taylor's improved shell. Nothing revolutionary per se, but it just looks more beginner friendly (ie you can go and edit existing expressions, rather than using shell history). Kinda reminds me of the only time I used mathlab.... some 13 years ago. Now if you could just change the prompt so you could easily pull out the code into doctests....

Laser guns, webcams, marshmellows, python and Halloween!

2007-11-01T00:00:00-06:00

My brother is the archetype of a hacker. We wasted many hours learning basic on the c64 back in the day. He used to hook up old monochrome apple monitors to stereo output and achieve some really cool visualizations. (Ben Harper's Opression is probably the best). He wrote tetris for the Tandy TRS-80 in an afternoon. He's created many potato guns and pressurized water rockets. I've been trying to get him to learn python for years but he has resisted. In the past couple of weeks he gave in.

The motivation? Cobbling a bunch of existing parts together to make a Halloween game for the trick or treaters to play. Here's the parts used:

MacBookPro with iSight
pysight
pygame
Marshie
Laser pointer mounted in pvc gun
Projector
Darkness

Next step, convince him to give a presentation at Pycon about it.

My brother's more detailed writeup is here.

Gnome devs too lazy for python?

2007-10-31T00:00:00-06:00

Some interesting thoughts about testing and python found among gnome developers. Joe Shaw was explaining how most new gnome apps these days are written in either python or C#. They choose C# for their app (Beagle) because "[they] wanted a strongly typed language" (I think he meant static typing but we get the point). Joe claims to have been burned by not having the compiler find "bugs".

His next two sentences are very telling:

Writing real applications in Python requires a discipline that unfortunately most people (including myself) are unwilling to adhere to, and this easily leads to buggy and hard to maintain programs. You have to be very diligent about unit tests and code coverage for every line of code, because you can’t rely on the compiler to catch errors for you.

If I read this with my cynical glasses on, I'm assuming that many java/c# developers think that testing consists of compiling the program. How sad.

In a rebuttal to Joe's dislike of dynamic typed languages, Thomas Vander Stichele claims that any language should have tests and coverage. And provides two reasons for it:

Confidence
Refactoring

Here, here, Thomas!

One wonders why all those redhat and canonical developers aren't on the static typed bandwangon for their apps, why weren't they distracted by the mono diversion?....

of OpenOffice, Symphony, KOffice, rst and python

2007-10-11T00:00:00-06:00

Just a few (rambling) thoughts on OpenOffice I've been having as of late. My daughter just started Kindergarten and my wife is now caught up in the PTA. So unlike the tech world, which tends to revolve around excel spreadsheets, the PTA revolves around word documents. My wife is relegated to using OOo. So far there have been no compatibility issues.

Calendars and OOo...

So instead of using something sensible like google calendar to create a PTA/School calendar that multiple people can edit (and actually do useful stuff with), the calendar is hand made in word. (Not necessarily hard, but painful, and very annoying for us automate the world freaks to watch being created. Plus you miss dates like the 31st. ~~Sadly, no templates are available for OOo and calendars.~~(see http://www.ooomacros.org/user.php#217442 ) I'm half tempted to rip apart the OOo doc and create an ical2odt.py But that would require people starting with a tool that generates ical...)

10 minutes with Symphony

I've heard about IBM's looming office application for some time, hearing that it was a thick client based on eclipse, with support for ODT. Well, when the beta1 came out last month I took it for a spin. The screenshots made me think that they had reverted to VisualAge technology....(it actually looks much better on linux) The download was pretty hefty at 220M, but the install was smooth. A minute or so after startup, I was able to create a doc. The view looked very much like OOo. So I started looking around it appears that they embed an instance of OOo (soffice.bin) as well as starting the java framework for the controller. All in all the experience was sluggish and appears to eat memory. I was hoping for an implementation in pure java/swt, but I was wrong. As such, I see no current reason to use Symphony instead of OOo. I'm not willing to learn a new gui, just to use more memory and end up with the same functionality. As an added bonus, it doesn't kill the soffice.bin instance when I quit!

KOffice

I've talked about OOo and KOffice before, and recently slashdot has too. Although web based processors are becoming more popular, I think anything more heavy than wiki editing is still best done on a thick client. (Note I'm not anti-AJAX, quite the opposite, as I've been playing around a bit (read full-time) with YUI and DOJO recently). So this is a prime area for KOffice to become the "firefox of word processors". And Windows/Mac compatibility is necessary to help in that effort (unless you believe the prime purpose of KDE is helping/forcing others to adopt Linux (which I've come to believe is a futile fight)). If KOffice can give 90% of people the features they need, thats a win for the ODT format. I'd use a snappier ODT client in a minute for most of my OOo stuff.

3 Some good bloat

But OOo is actually really powerful there are features lurking under that bloat that just might come in useful (about once every 3 years or so). I found out about the form creation the other day. If you export as PDF, then the form will be editable in pdf! Cool, and somewhat useful depending on who you and your audience are.

So what to do?

Since my preferred lightweight markup these days is rST, I was thrilled to see this rst2odf converter. While I'm enjoying rst2s5 I keep thinking that sometimes an rst2impress or rst2prosper could be very useful at times. Or now that I'm fiddling with dojo, a rst2dojopreso, and perhaps integrate my pygments svg output....

Ok, that's enough rambling, just note that most of the power of OOo is in the format standardization. It allows one to create ancillary applications and scripts easily. It permits us to sleep easily at night knowing that we will be able to access our data in the future.

Slicing web pages with inkscape (and python)

2007-09-28T00:00:00-06:00

Using the Plugin

Was watching this video describing how to "slice" web page designs two days ago and it inspired me. After a tiny bit of digging I found that one can easily script inkscape using python.

Here's the process I've used for slicing web layout with Inkscape. Create your webpage layout (set page units to "px", width/height appropriately and snap to 1 pixel intervals. This should allow pixel perfect alignment). Then create a new layer, naming it slices. Draw rectangles over the areas you want to slice (set x,y,width,height to whole pixel values). Name these rectangles using the Object Properties found in the right click contextual menu (the saved images name will be based on that value, so name them something like "header" instead of the default/non-useful "rect4312"). This is all explained in the video above.

What the plugin then does is iterate over all of the rectangle definitions found in the slice layer and set the opacity of the slice rectangle to 0. (That allows you to make slightly transparent slices, which are easier to deal with than invisible ones) It then creates pngs for every slice. After completing the slicing, it then sets all the slice rectangles to red at 25% opacity. To continue working on your design without having to deal with the slices being selected, just "lock" the slice layer and your clicks should fall through. You can also click the "eye" on the slice layer to not view it.

Oh yeah, I guess one should install the plugin. On gentoo I need to put the .py and .inx file into /usr/share/inkscape/extensions, then after a restart of inkscape the plugin will appear in Effects->Export->Export Layer Slices.

Creating an inkscape plugin in python

Scripting inkscape was pretty straightforward. I did it by looking at the other plugins and not reading (or knowing about) the script wiki entry I listed above. The basic idea is to create a class that subclasses inkex.Effect, and implement an "effect" method. The .inx file is a simple xml file to designate where in the gui your plugin appears as well as provide for input required by your plugin.

update - Code is now up on github

Howto create a pygments backend

2007-09-05T00:00:00-06:00

Pygments is a source code highlighter. I believe the main purpose is to format code/markup for webpages (forums, blogs, wikis, trac, etc). (I used it on my previous post). But it also has renderers for latex, rtf and bbcode. I happened to desire an SVG backend yesterday, so I'll share my experience.

On first thought I went to #inkscape (an svg editor) and asked if they understood (or could import) the html/css output of pygments. It turns out that with some careful substitutions it can be done. One kind soul even went so far as to write me some java(!?!) code that did this. Finally my brain kicked in and I though why not just write an SVG backend for pygments? Pygments is written in python and I have done some SVG creating using the SVGdraw.py code before that was pretty straightforward.

(Seems like the creators of SVGdraw.py have let it drift into the ether.... Sad cause it is a useful tool, luckily I have a copy)

So I asked about on #pocoo how to create an SVG backend, and was told to model it after formatters/html.py. Here's what I learned while creating the SVG backend:

Subclass formatter.Formatter
Implement a format(self, tokensource, outfile) function. Tokensource is an iterable of tokentype, token tuples. Using style information (mapping tokentype to a style) found in the self.style.styles member, write your formatted content to the outfile
Add some class attributes: name, aliases, and filename used for command line help. Mine were "SVG", ["svg"], ["*.svg"] respectively.
Update formatters/_mapping.py to add your formatter to the FORMATTERS dictionary.
This is the step after the last step

Try out your new code by running the following command:

./pygementize -f svg fileToMakePretty.py

(replace svg with your new formatter)

Boilerplate for maintainable, distributable, testable python code

2007-09-04T00:00:00-06:00

There's a difference between code when you know you will be using it later and code that you're going to throw away. There's also a difference between being lazy and taking time to follow best practices. Below is an example of two command line programs that do the same thing. One is 3 lines long and the other is 100. (Of course the example is contrived, and usually the logic will be more than 3 lines. For cat I would usually use the command line cat) The effect of having the actual logic portion be small is that this is boilerplate code useful for starting python command line programs.

The extra length needs to buy us something. Here's my ideas of what it gives me:

Order - I know where items belong
Isolated Functionality - No (or limited) execution when importing/reusing
Logging - rather than dealing with prints that clutter the screen
Testing - I've got doctest built in, but I have a unittest file as well that gives decent coverage
Documentation - Illustrates module and function docs as well as doctests
Command line docs - use of optparse
__main__ - Illustrates use of main statement (with argument arg for reusability So lazy web. Am I a pep8 following geek that just complicates things? Should boilerplate code include more? less?

I'm asking cause I'm doing two intro to python classes soon, and I thought that I might use this as an example since it covers a bunch of ground.

Throwaway Code

import sys

for line in open(sys.argv[1]):
    print line,
More complicated, yet testable, maintainable

#! /usr/bin/env python
"""
canonicalcat is an example of writing a python program that perhaps
you want to distribute or will be maintained over time.  As such one
should add documentation and this is an example of module level
documentation.  If one were to import canonicalcat and type
`help(canonicalcat)` it would spit this out
"""

# These are the imports from the python stdlib
import sys
import logging
import optparse

# Some like to separate additional libraries from the standard ones


# file meta data
__version__ = "0.1"
__author__ = "matt harrison"
__license__ = "psf"


# GLOBAL ARGS/CONSTS
# Since python doesn't have constants, we can emulate them
# using (naming) conventions.  Put them here


# GLOBAL INIT
logging.basicConfig(filename=".concat.log", level=logging.DEBUG)


def cat_file(fin, outfile=None):
    """
    Main logic to cat file

    >>> import StringIO
    >>> fout = StringIO.StringIO()
    >>> cat_file("small.txt", fout)
    >>> lines = fout.getvalue()
    >>> lines
    'foo\nbar\n'

    """
    logging.log(logging.DEBUG, "CONCAT: %s"% fin)
    if outfile is None:
        fout = sys.stdout
    elif isinstance(outfile, str):
        fout = open(outfile, 'w')
    else:
        fout = outfile

    for line in open(fin):
        fout.write(line)


def _test():
    import doctest
    doctest.testmod()


def main(prog_args=None):
    """
    A main function.  Rather than putting this logic into the the if
    __name__ statement below, creating a main function allows other
    programs to use main logic.  It also allows for testing, by
    passing in args rather than monkey patching sys.argv (which won't
    be thread safe).

    >>> main(["canonicalcat.py", "small.txt"])
    foo
    bar

    """
    if prog_args is None:
        prog_args = sys.argv

    parser = optparse.OptionParser()
    parser.usage = """A python implementation of 'cat', default use is to
    provide a filename to cat"""
    parser.add_option("-o", "--output-file", dest="fileout",
                      help="specify file to cat to (default is stdout)")
    parser.add_option("-t", "--test", dest="test", action="store_true",
                      help="run doctests")

    opt, args = parser.parse_args(prog_args)

    if args[1:]:
        cat_file(args[1], opt.fileout)

    if opt.test:
        _test()
    else:
        parser.print_help()


if __name__ == "__main__":
    # when one "exectutes" a python program, it's __name__ is
    # __main__, otherwise it's name will be the module name
    main()

Does Oracle really love Rails?

2007-08-01T00:00:00-06:00

If one searched for Oracle and Rails they're bound to turn up a bunch of results that appear to indicate that Oracle is on the Rails bandwagon. How Web2.0 of them. (I'm trying to think of some funny snide remark about a rails acquisition, but can't really....)

Apparently someone failed to tell Oracle that the IDE for Rails, TextMate only runs on MacOSX. And MacOSX only runs (legally) on Apple Hardware. And it's been what almost two years now that Apple hardware went to x86. Oracle DO YOU KNOW THAT?

Ok, the purpose of this rant is to show how 2 sided (or out of touch) Oracle is. If you happen to need to get the Oracle Instant Client for Mac OSX to work on a MacTel machine (say if you need python or ruby or perl to access the database, or even to do Rails work), you are out of luck! Sadly it doesn't indicate that the downloads are PPC ONLY! (NOTE: This is no longer the case ;) ). But a little more digging will show that Oracle isn't planning to release drivers until after 11g comes out. (I really do feel sad for the OTN guy trying to make the MacTel users feel like Oracle actually cares (or listens). I don't envy his job.) How hard can it really be to make the drivers work on x86? Had this been an open source project it would've been done years ago. (Or I could go fix it at least). Here with a proprietary vendor, I'm at their mercy, and I can't do anything about it! How un Web2.0 of them!

(The other purpose of this rant is to note that getting python(sqlalchemy) to talk to Oracle from MacTel is a no go right now)

OSCON 2007 (July 25)

2007-07-25T00:00:00-06:00

I arrived in Portland last night for OSCON. From a conference standpoint, it feels very similar to when I attended in 2005. The attendance doesn't appear to have grown much and the exhibition hall is about the same size. Though there appear to be a few more "community" exhibitors than previously.

As far as hardware goes there appears to be an abundance of Apple laptops (probably around 70%). Of the remaining it seems that a good 2/3 of them are running windows. Perhaps I'm too much of a Linux bigot, but I find it amazing that a conference that Mark Shuttleworth keynotes at, that is name "Open Source" conference, that most attendees are using proprietary operating systems! Oh well, I digress.

Keynotes

The keynotes this morning were pretty good. They were of the short/sweet variety. Each person talked for 15 minutes only. Tim O'Reilly started off by talking about how to see the future of technology (watching alpha-geeks). He also talked about how web 2.0 is removing some of the openess of the web. Data lockin and removing the ability to "fork" networked applications were some of the issues he mentioned.

Some Intel folk did a clever trick by dressing up in a suit and tie and pretending to pitch/sell their new Threading Building Blocks library. Then they ending saying they were open sourcing it

Continuing on the threading thread, an MS researcher Simon Peyton, explained Transactional Memory and the basic building blocks, "atomic", "retry", and "orElse". After he explained these, their need and the problems they solved he indicated that they been implemented in one of his projects, Haskell for over a year. Should be interesting to see constructs such as these gain adoption given that Moore's law seems to be relegated to multicores these days.

Tim then interviewed Mark Shuttleworth. The main point I gathered from their 15 minutes was that Ubuntu was really successful in "innovation in and through collaboration". Their Launchpad tool is an example of hub in a collaboration engine for end developers as well as upstream component providers.

SOLR - Kimsal

I went to a session on Solr that demoed setting it up and adding some data to search engine. Solr is a standalone search server that provides an easy resty interface.

What does "Open Source" mean?

This was an interesting panel led by Danese Cooper. At issue was SugarCRM and Socialtext's use of attributions licenses that weren't approved by OSI.

Two announcements were made, one that non-vanity license, CPAL for attribution was being accepted by OSI (after SocialText went through the ringer for it), and that SugarCRM is going attribution free by adopting GPL3 for their V5 project.

There was a lot of discussion about the impracticalities of placing a large amount of attribution text in small gif ad's on web pages of derived projects.

Hadoop - Doug Cutting

Hadoop is an open source implementation of Google's MapReduce toolset in java. Originally it was part of the Nutch search engine, but it has now been pulled out and is mainly developed by Yahoo.

The basic idea is that to scale to petabyte datasets your data will not fit in memory. You need to put it on disk and that can be problematic since disk seek speeds haven't improved much over the years. So to deal with this you build a cluster of machines that can churn through data at maximum disk i/o.

Yahoo, for example, is using Hadoop to perform queries on their large dataset. These are batch mode queries and some are queries that weren't really thought of when the data was placed on the disk. The MapReduce paradigm allows for this kind of adhoc querying.

Foundations of Open Source - Waugh

I then went Jeff Waugh's talk about the foundations of opensource. His belief was that too much focus on open source licensing has led many to neglect the finer and more important points of what it really means to be "open source", such as community and governance. He then described his experiences with Ubuntu and the Gnome project. Ubuntu was lucky enough to start on the shoulders of giants, other open source projects that had been in existence, and had structured itself to avoid some of the hurdles that these projects ran into.

He was using Clutter for his slide show (fancy open-gl), which I hadn't heard of previously.

Atom Publishing Protocol - Gregorio

Joe Gregorio presented on the Atom publishing protocol. Through some nifty deception he was able to guide simpletons through some of the finer complexities of the protocol.

Generating Gorgeous Word Documents - Koziarski

I liked the methaphor used during this preso. He compared document generation to web mockup generation. Basically the designer gives the developer some html layout filled with static text, "Lorum impsum, foo..." and the dev will then leave the pretty formatting but replaced the static elements with the appropriate business objects.

Koziarski needed to create doc and pdf files, but using COM/MS Office APIs wasn't an option (need to run on linux), rtf, latex and pdf generation also didn't work because the designers couldn't visually "design" with them. The solution was to have designers create documents in Open Office, and then stripping out their static text in the resulting XML.

Because they ran into issues about OOo requiring X11 on linux (headless X servers gave errors), they ended up complicating the process by throwing Amazon's S3 and SQS into the mix.

All your comments are belong to reddit

2007-06-18T00:00:00-06:00

I guess this has been around since the dawn of site aggregation (ala slashdot). I'm just sort of frustrated that my previous post's most valuable and relevant comments weren't posted on my blog but on an aggregator of programming links (reddit). I'm not implying that reddit isn't useful for finding interesting fodder every so often, it is just annoying that relevant comments to your post have a further degree of separation your post. So unless you have someone nice who actually bridges the comments to your post(like Fredrik) you might never see those useful posts.

[end rant...]

Silly Python is_nan implementation

2007-06-15T00:00:00-06:00

So [apparently equality for NaN doesn't make sense](http://mail.python.org/pipermail/python-list/2005-July/330042.html_. (NaN or Not a Number is division by zero, and other other operations that return non numbers, which is different than infinity or negative infinity). That makes sense, but sometimes you want to know if a number is really not a number. So how does one do so?

Javascript happens to have a nice isNaN function and today I happened to need one in python. My naive implementation didn't work (is_nan1). But if you coerce to string and compare against "nan" that works:

matt@r52 $ python
Python 2.4.4 (#1, May 12 2007, 23:48:56)
[GCC 4.1.1 (Gentoo 4.1.1-r3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> a = float("NaN")
>>> b = 0
>>> c = float("nan")
>>> d = float("infinity")
>>>
>>> def is_nan1(num):
...     return num == float("NaN")
...
>>> print is_nan1(a)
False
>>> print is_nan1(b)
False
>>> print is_nan1(c)
False
>>> print is_nan1(d)
False
>>>
>>>
>>> def is_nan2(num):
...     return str(num) == "nan"
...
>>> print is_nan2(a)
True
>>> print is_nan1(b)
False
>>> print is_nan2(c)
True
>>> print is_nan2(d)
False
>>>

Update math.isnan is available

Visualizing harddrive space under Linux

2007-05-30T00:00:00-06:00

Today, my laptop disk became full. So, I wanted to get the most bang for my buck and not waste my time erasing small files that weren't taking up a bunch of disk space. How does one do that? One way is the command line utility du, but I knew there were a few programs out there that visualized the amount of space used. (Plus I knew that du would send me digital reams of information, so I figured visualization would be easier to digest).

Searching led to this article and from that I installed KDirStat and filelight. (Thankfully they were both an emerge away thanks to Gentoo.)

How do the two apps compare? I think treemaps are very useful for many visualizations. But in this instance, the radial interface of filelight come across much simpler than KDirStat. KDirStat doesn't provide any textual information in the treemap, but instead relies on a treetable control in another pane that shows all of one's files. If you click on a file, it highlights it in the treemap, and vice versa. Filelight, on the other hand updates the labels based on the band you are mousing over. When you hover over a ring, it indicates how much disk space that ring is taking and it's size (ie: /home 12,499MB 40%). Also, when one mouses over a ring, all the child directories are given labels. This interfaces provides a quick way of seeing what directories are taking up all the space. It is quite intuitive and I quickly discovered a few things:

My 20 copies of the kernel (linux-2.6.15-gentoo-r1/ through linux-2.6.20-gentoo-r8/ were taking some 8.5 gigs in /usr/src. DOH!
~/.googleearth has some 230MB of cache files!
Hugin (panoramic stitcher) works in TIFF (~400MB of data files from attempts to make 3 panoramic photos) Using the information these tools provided me (and copying some mp3s to an external drive, quickly gave me back some 14 Gigs).

To be fair, KDirStat also shows this information, though it only shows parent folder sizes in the treetable, while the treemap itself shows individual files (clicking on a directory draws a red square around the files it contains). It also appears that the backend for KDirStat is a little smarter than filelight. Filelight rescanned directories upon double clicks (which can take a while on slow laptop drives), which KDirStat appears to cache and allows you to quickly zoom in and out.

Would I use these programs everyday? Probably not, though they were quite useful in cleaning up the junk that was taking the most space. Had I not used them I probably would've forgotten about all those old kernels, and I would've never known about the google cache.

Below are screenies of both programs. KDirStat only scanned my home directory while, Filelight scanned the whole "/" drive (the blue rings are my home directory).

May Utah Python Meeting

2007-05-11T00:00:00-06:00

Last night I attended the Utah Python User Group meeting. The meeting location was changed to a library in Salt Lake instead of the Mozy headquarters in American Fork. It appears that the more central location was quite a draw since there were some 20 people in attendance.

Jonathon Ellis spoke on Python 202. He went over the various features found in 2.x series.

I spoke on Code Standards and Complexity for Dynamic Languages (with some examples in python). To any perl or ruby folk [lazy web] out there, if there are tools similar to the python examples, I'd love to hear about them.

Note that I've blogged briefly on code auditing in python previously.

Goodbye Beryl, Hello 24 Incher

2007-05-08T00:00:00-06:00

I got a nice 24 inch Dell flat panel monitor for work. Here's the good: * Bright colors (my laptop monitor looks dull now) * Wide (can comfortably browse two websites side by side) * Gentoo made installing drivers painless * xrandr 1.2 is nice * X developers rock

And now the bad: * I don't know how I can ever go back to anything smaller. It just seems so "normal" now *Beryl (accerlerated X won't work with dual screen)

Re: installing drivers, I've messed around with screen cloning and dual screen before the xrandr 1.2 days and it was a pain. Well, when I got the monitor xrandr 1.2 was not yet released. Gentoo made it easy to upgrade the drivers. And when new betas were released and the tree wasn't updated it was 3 commands for me to update (cp the ebuild to the new version, ebuild digest and ebuild newversion).

Re: Beryl. I'd been growing quite accustomed to beryl, despite its dislike of java apps. When I'm in dual screen mode, the screen is too wide for accelerated X to work. According to Keith Packard this can be coded around (as is done for Macs/Windows drivers with the intel chipset). I'm not sure if the work is scheduled at the moment. But I much prefer dual screen to no beryl....

And now some notes showing how xrandr1.2 is (for people who want to give presos or whatnot):

With the 24 inch plugged into the vga port of my laptop, it automatically comes up in cloned mode (Though the virtual size of my laptop is the physical size of my vga monitor....).

To enable dual screen run a command like this:

xrandr --output LVDS --left-of VGA --auto

To turn off the VGA monitor:

xrandr --output VGA --off

To go back to "clone" mode:

xrandr --output VGA  --auto

Exciting Times Ahead

2007-03-28T00:00:00-06:00

The last couple of weeks have been stressful, bittersweet, and exciting. An exciting opportunity fell upon me and the end result is that I resigned from SpikeSource and am now starting up a company.

SpikeSource has been wonderful to me, and I've very much enjoyed my time there. I've worked with startups ever since graduating, but when I joined Spike there was 14 people in Kleiner Perkins incubation site on Sandhill. Things were wild and crazy and busy. There was a cool anticipation in the air for the future. Over time, I was able to work very closely with the founder, Murugan. I was also able to see the corporate side of Open Source. I worked with commercial companies, commercial open source companies as well as non-commercial open source entities. I was also able to attend many conferences (oscon, pycon, mysqluc, linuxworld, osbc, foss.in) and take a trip to India. I got to program in python, a lot. I got others to program in python, a lot! And I got to play cricket a couple of times.

But the best part was the people I worked with. Every startup prides themselves on their people. And Spike was no exception. The people were wonderful and I learned much from them, and will miss them. Thanks again Spike!

Interesting mashup of Youtube, REM, Yoko Ono and Political Activism

2007-03-12T00:00:00-06:00

I heard the following cover of #9 Dream by John Lennon in the car today by REM. Interesting how a cover makes you appreciate a song more. I don't mean that in a bad way, I actually really liked the REM cover. So I went home and found this. The "video" is done in an amateurish, yet interesting way to get across their point.

That in turn led to this, which seems to be the best thing Ms. Ono has done since breaking up the beatles.

A few questions: First, what the US do about Darfur? (Probably very little since there don't seem to be any weapons of mass destruction [read oil] in the region). Will the actual downloads be through iTunes? Has Youtube disrupted iTunes? Ok, perhaps not the ipod, but with Youtube I can listen to most songs on demand (as well as see a video of them). Does youtube disrupt Karaoke as well?

In Russia Open Source Rescues You

2007-02-28T00:00:00-07:00

Given that MS is cracking down on poor Russians (which is weird since rampant piracy of their products only serves to promote their monopoly), I would've thought that some open source vendor would jump in to rescue Mr. Ponosov (head of school master in rural Russia charged with stealing software).

Seems like RH, Novell or Canonical(?) would come in and say, "We'll pay your $10,000 in fees to MS but will also give you free software for your whole school district." Sounds like some good cheap marketing once the news people pick it up....

(The Steve could one up everyone and say not only does he hate DRM, he supports free (gratis) software.... as long as you buy the hardware)

Interesting turn of events on gentoo-dev

2007-02-27T00:00:00-07:00

How many times does this happen? You come back as a developer to a project you founded? That's gotta take some humility. It's my please to introduce to you Daniel "drobbins" Robbins. Daniel is going to work with the amd64 arch team but will probably venture to other areas too. Daniel doesn't have much experience with Gentoo so let's give him a helping hand in the start.

Will be interesting to see what kind of role he takes.

Pycon2007 observations and thoughts

2007-02-27T00:00:00-07:00

They say you know a foreign language pretty well when you start dreaming in it. It's a pretty surreal experience at first. Well, what about when you dream that you are sitting listening to conference talks and the conference is over? (Which happened to me last night) Probably means you just have conference overload. Being in talks, bofs, and keynotes from 9 am to past midnight for about 4 days straight can do this to one I assume.

One of my favorite things about Pycon was just seeing what other really smart people are doing and absorbing what I can. It's very inspiring, while at the same time exhausting. Most of the tutorials/talks/keynotes/lightning talks were pretty good. I've posted my notes to some of them.

Ubuntu appears to be the linux distro of choice now. I think I was the only one running a non-ubuntu linux (gentoo). This was quite surprising cause I met quite a few last year running Gentoo. (But since both make pretty liberal use of python I won't complain too hard). Only saw one Vista machine (Jim Huginin), but the rest seemed pretty evenly split among mac/xp/ubuntu. Draw whatever conclusion you want from that.

It appears that a lot of companies are looking to hire python people, and are having a hard time finding them. (Perhaps this is an advantage to the people who do virtual teams, like Canonical. Say what you want about telecommuters, but they don't have to limit talent to a geographic location).

Bofs are cool. (Birds of a feature sessions. A board is placed with times and rooms where people can put any random topic they want to talk about on it. People just show up and discuss things). They allow for more interaction and thus I think more details. I didn't attend too many last year so I guess I missed out then. The ones that stood out for me were on twisted, OLPC, Bazaar and advocacy. I was just a fly on the wall for one of the twisted sessions (have tried to learn twisted once or twice but given up after hello worlds), but it blew my mind how much information some people (glyph) can hold in their heads. During OLPC I got to play (or tried to play) with the laptop for a bit. The design of the hardware looks impressive, but the experience went downhill after that. The software was very alpha, and I had a hard time getting it to do much. (Hence the many impossible deadlines they still have til they ship in the end of the summer). If you know python, go help them out. The advocacy bof was interesting as well. Interesting to hear others (small and big companies) issues with adopting python as well as possible solutions (marketing and supporting local user groups seems to be the two biggest). The Bazaar stuff wasn't too new for me, but it was interesting to hear what Canonical is doing with launchpad. Also all/most? of the work Canonical does is in python.

Pycon 2008 is in Chicago. I've never been to the windy city before. Hopefully the site is big enough to hold lots of people. Attendance jumped 40% this year to about 600. Perhaps they can break 1000 next year.

One last hint if you go next year... check out #pycon (it can be quite amusing, yet also helpful at times)

Why Apple really doesn't care about DRM

2007-02-26T00:00:00-07:00

Slashdot reports on Apple's anti DRM again. Sadly none of the posters seem to understand why Jobs really doesn't care if Apple has DRM'd content or not. My brother explained this to me the other day and it was so obvious I wondered why it isn't more common. (See defective by design).

What does apple sell on iTunes? Audio in the .m4p format. Which is simply DRM around the AAC format. Now here's the quick question: How many portable devices play m4p? Well ipods (and itunes but let's ignore the computers) do and ummm, no one else. How many portable devices play aac format? Ummm, ipods and .... nope no one else. So even if Apple sold DRM free content, you would still need an ipod to listen to it. (Even if aac content was sold it would probably be a year or so til firmware supported it in non apple hardware).

Proving once again that Apple is a hardware company. They are out to sell [read lock you in to] their hardware. So it's hard to tell the sincerity of Steve Jobs, when this could really be just a ploy to appear to come down on the side of the consumer, when really you are still locked in to their platform (oh and they sell more ipods since you'll need one for the DRM-free music).

[disclaimer] my wife owns a nano... we have yet to visit itunes music store

Lessons in Open Source from the Django project [pycon]

2007-02-24T00:00:00-07:00

I feel a little bad for just posting these notes. But I think they can be useful for others. These are notes learned about running an open source project from the Django project This talk was by Jacob Kaplan Moss.

Arguments for open sourcing

Money- consulting, sell proprietary add ons
Hundreds of people will contribute (if compelling product)
Self improvement model - Very clean because of peer pressure/review Django is light years ahead, if it had stayed closed source
Geek cred - Will be able to higher better, talk at conferences, be more recognized
Moral argument - We've built a business on top of open source. It's time to give something back. But where do you draw the line?

How did this work when approaching management? Surprisingly the moral argument held the most compelling to the managers.

Selling open source to other companies

Unfortunately FUD from MS has many people predisposed against open source. People refused to run Linux because it was "anti-capitalist" (literally).

Most effective counter argument to communist threat of open source was to talk about freedom. Freedom from lockin. Your data belongs to you.

Lessons for maintainers

We aren't perfect or great, probably just a little better than average.

Build a community, not just a mailing list. Be very conscious of tone. Followup with constructivly with criticisms on blogs (ie "Thanks for you criticism, how can we do this better?"). [matt] probably comes from journalism background
Be careful of trolls, and just ignore them. Possibly ban them.
Spam can't be an afterthought. Don't think oh, we won't get spammed. Google groups does a good job.
Listen to the community. Everyone has good and bad ideas. Don't pay too much attention to vocal minority. But be willing to take big risks when the community asks for them.
Have to be willing to ask for help and delegating things.

Testing Tools Panel [pycon]

2007-02-24T00:00:00-07:00

I've blogged about bugs and testing in open source previously, I was quite interested in this panel. I was surprised because there was little discussion of code coverage, because I think it is quite important for dynamic languages to have good coverage. (I find that doctest and coverage.py not working together is a huge warning sign that people are ignoring coverage). Anyway regardless of my rant, here's a summary of the testing panel. Moderated by Grig Gheorghiu of Testing tools wiki

Benji York: zope.testbrowser
Brian Dorsey: py.test (represent Holger Krekel)
Chad Whitacre: testosterone (created) switched to Nose
Ian Bicking: paste.test.fixture, minimock, FitLoader
Jeff Younker: PyMock
Kumar McMillan: fixture - module for loading and referencing test data
Martin Taylor: test framework within TI
Neal Norwitz: PyChecker
Tim Couper: WATSUP (windows GUI Testing)
Titus Brown: twill, scotch, figleaf, pinocchio

Intros

BY: testbrowser not currently Zope-centric. Useful for functional tests. Built on top of mechanize. Important that you can read the test and tell what it is doing.

BD: py.test, minimal api. Goal: no api, just use assert. py.test now has a release, previously only svn possible. Automatic test discovery discovers, full doctest.

TC: Absence of windows testing tools. Make tests easy to run from a gui.

MT: TI has silos of test automation. Try to standardize on a testing language using python. Replace Testdirector with python api to talk to (Testopia, testlink... or one he hopes to write). Has approval to release as open source. Hoping to release some time this year.

KM: Building an ETL system in python. Made a module called "fixture" which loads in data sets at the beginning of tests. Colleague wrote "nose" (Jason Pellerin).

IB: Was interested in Fitnesse testing. Write fitloader, then decided that fitnesse was stupid too. Minimock was a test to see how small he could get it. I hate unittest and want it to leave the stdlib.

NN: Testing should be done to prevent bugs. Pychecker tries to find bugs. Work at Google is to promote testing internally.

JY: Mock objects let you separate your units of code from the rest of the program. Not easy to mock out results of generators. Still a lot of work to do with it, error checking, reporting, playback utilities.

CW: Testosterone is manly testing interface. I'm addicting to dots. Attempt to put some sanity on top of it.

TB: Twill is similar to zope.testbrowser and paste.fixture. Built on mechanize. Useful for testing and monitoring websites. Doesn't have zope or wsgi in the name anywhere.

How do you approach testing in your own projects? Unit, functional, GUI? What provides the most payoff?

BY: Write tests and prose first (documentation driven design). One day the bullet whizzed passed my head, and I've been addicted since them.

BD: Unittesting. Specifically test first.

TC: Depends on where you are in the organization.

MT: Most effective is API testing. Above unit testing. Around 2 million testcase.

CW: That's A LOT OF DOTS. (laughs)

KM: TDD made sense at first but made my head hurt. Makes sense to write functional level test first. Some of our code doesn't have unittests because we test at a higher level.

IB: I've sometimes tested too much. Lots of low level, and very few high level. I hate that you can loop over strings. (Don't know if it's a list sometimes)

NN: Lots of unittests like Ian. Need some level of system tests.

JY: Don't do much function or integration testing at my work.

CW: Tried TDD but didn't convert. Testing mindset different than design and coding. I enjoy more being in the creative design mindset. Will get code out first but later put in unittests to satisfy the dot addiction.

TB: Don't agree with Chad. Kevin Kelley, Fast, Cheap and out of control. Describes coding and testing style. I don'd do TDD, I do Stupidedly driven testing. Just test.

IB: I wanted to test for a long time and hated the tools. Once I found (built) the tools, I was happy (and enabled to test).

Confessional: Is there a recnet example when you didn't live up to your testing standards?

TB: I don't have standards.

CW: More important to code and get your ideas out than test.

JY: All the time I don't meet my standards, then the code slaps me hard and I have to go back.

NN: Dashboard at Google for test results. Only unit tests were written no system level. And that's sufficient. Be pragmatic and do what you need.

IB: I don't test well when I don't understand the problem I'm trying to solve. Really need to write it once, test and then re-write if you don't understand problem.

KM: Most important thing is having a regression test suite. You know when you break something.

MT: Confession, I don't write enough tests for my test framework

TC: Haven't talked about manual testing much, but it's very useful.

BY: If you are prototyping you probably don't need to test. When I'm in "hack" mode, code doesn't get tested.

Name one testing tool you use

BY: Zope.testing, also not zope.testing

BD: py.test, doctest

TC: py.test, doctest

MT: Abbot (for java swing)

KM: Buildbot

IB: py.test, nose, twill

NN: unittest, but I don't like it

IB: unittest, idea and eclipse, refactoring in java is nice

CW: nose

TB: nose, buildbot, paste.lint

Web Framework Panel notes [pycon]

2007-02-23T00:00:00-07:00

Here's some notes from the web framework panel at Pycon, discussing various attributes of the frameworks and why python tends to have so many frameworks.

Discussion led by Titus Brown featuring:

Zope - Jim Fulton
CherryPy - Robert Brewer
Django - Adrian Holovarty
TurboGears - Kevin Dangoor
Pylons - Ben Bangert
Nevow - Duncan McGregor
Spyce - Jonathon Ellis
Pyjamas - James Tauber

Why are there so many web frameworks?

JF: Leading framework when Zope started was CGI. Zope started out as an application

Duncan: Nothing worked with twisted.

Robert Brewer: CherryPy scratches a smaller itch. Be small enough for devs to keep in head, but also to allow end users to scratch their own itch. Tries to only be http, not tempating or db.

Kevin Dangoor: Started a one man company (end 2004). Wanted to write as little code as possible. Every part of ROR was already in python, just not integrated. Messed around with different components and settled on his stack.

BB: Pylons started before django and tg were available. Didn't want to maintain anything. Re-use was a key factor. Used Zope and Plone previously.

Adrian: Django extracted from a real world site. Needed to be web apps really quickly. Turned into conventions, into framework into open sourcing.

James Tauber: (Pyjamas) Was using/creating(?) a framework (leonardo) but spent too much time, so switched to django.

Jonathon Ellis: Only real thing available at the time was Zope, which scared him. Got hired using Spyce, and then started maintaining it. When you control the whole stack (via django) you can innovate faster.

If spyce (your framework) died would you re-create it?

JE: I didn't start a framework, I revived one. I like what django does with innovation. But their ORM is feeble.

DM: Perception that the multitude is a weakness. Should turn it on it's head. A sign of intelligence. .... But I'd shoot myself if nevow wasn't around.

RB: Go ahead and build your framework if you want to.

JT: Re-inventing the wheel is great if your goal is to learn more about wheels.

DM: For Content Management I'd use Zope or Django

KD: Pylons is pretty cool

BB: I like TG

KD: More innovation where the individual pieces have thier own dictators. Pushing to break TG code into smaller projects with their own leaders, which will lead to more innovation.

RB: Make a cherrypy session module that works well.

Tightly integrated vs loosely coupled?

Titus: Django claims to be tightly integrated but loosely coupled. What does that mean in english? (Please use wsgi in your answer)

Adrian: You can't have it both ways. Try to make the django site very approachable (even for people who don't know python). Huge turn off to say, "You can do it this way, ps you can also do it these 3 other ways". Need to pick your focus. Where we've failed is that we haven't done the advanced tutorials (for people who want to use some different components) (Full stack is good for beginners)

Jim Fulton: You have to have documentation. Stories for people getting started and stories for solving them. Peeve, understand the appropriate use for technologies. Zope isn't a replacement for simple CGI. Need to be clear about goals and document how to solve those goals.

Titus: Pylons says on the front page "Here are the choices you need to make"

BB: Probably need to change the front page. Have a default suggested component. Have something that's easy to use, but powerful enough that if won't hold you done later.

Python virtue: One way to do it? Web frameworks are perlish

RB: These frameworks are domain specific. Different domains have different solutions

Ajax and javascript

JT: For pyjamas, TG or cherrypy would probably be a good fit (even though I use django)

Titus: Is the web about Pages with ajax?

JE: Are two types of web pages flex/openlazlo vs page based.

AH: We don't ship with JS framework. "Having python produce javascript is kind of like using a wheelchair since you're too lazy to walk". Django has a bad rap that "it doesn't do ajax", but you can use AJAX with it just fine. We need better documentation, here's how you can use js with django. We have pretty good docs, that the most important thing you do.

JF: Plone is using KSS. Zope corp uses Mochikit. It isn't clear what the relationship between server techs and js should be.

DM: We were there first. We had AJAX before it was called AJAX.

Titus: Is that documented?

(laugh)

BB: Ported rails helper functionality to python. Which generates some js using scriptaculous.

Arches in the Winter

2007-02-20T00:00:00-07:00

We were able to sneak away for the weekend and took the kids to Moab, Utah. Home to wonderful mountain biking, rock climbing and scenery. We didn't have time for too much, but we were able to get into Arches National Park

I'm always amazed at the diversity of geologic wonders one sees there. While we were pulling into the city, the alpineglow of the sunset made for spectacular colors. Sadly I didn't take any pictures. There appears to be rock of every color there: red, orange, yellow, white and green. With the desert varnish, blue and purple seem to come out as well.

I did take some panoramic shots to play around with. I believe one series has over 7 arches visible in it! (Not too surprising, given that there are over 2000! arches in the park)

We passed through Price, Utah on the way and Olivia was itching to see the dinosaur museum. There was also an exhibit by B. A. Grimes showing some great photos of the San Rafael Swell and Nine Mile Canyon.

Professional Use of iPods

2007-02-01T00:00:00-07:00

Just read an [article about using iPods the help pitchers in the Majors[(http://sports.espn.go.com/mlb/hotstove06/columns/story?columnist=stark_jayson&id=2744070). The main idea is that a pitcher can watch a video of every at bat with the opposing team members and see where they should throw the ball. And they can do this in between innings.

Sounds like the low tech version of what my dad used to do when he coached little league. He used to scout the other teams and write down where everyone of their players hit the ball. Then when we played the team, before each player batted he would adjust the position of our defenders as necessary. It worked quite well as I recall

Seems like there are all sort of visualization opportunities for ipoders. Sell videos of rides at amusement parks. Sell videos of people swinging their golf clubs at the range....

resume scp after interrupted downloads (use rsync)

2007-01-24T00:00:00-07:00

If your download got cut off using scp , stop and read this before using scp again!

Telecommuting has it's perks. But one of the downsides can be the network issues. Especially if you have to download large amounts of data often. Having been bit by the interrupted downloads one too many times I found the following solution here.

The problem is that scp doesn't support resume, however rsync does. So create the following alias and you should be good to go:

alias scpresume="rsync --partial --progress --rsh=ssh"

Perks include understanding the same user@host:path syntax as scp as well as being able to resume a broken scp download (Note: I'm not guaranteeing this, but after downloading 90% of 400MB I was able to pick up the rest using this).

Web 2.0 for a Church???

2007-01-19T00:00:00-07:00

Last night I attended the "Tech Talk" for the LDS Church (aka the Mormons). A church sponsoring a tech talk seems sortof like having google workshop telling you how to achieve nirvana. For those uninitiated with the LDS faith, obviously there is a huge interest in genealogical research and making that accessible to the masses. And there are actually huge technology problems around trying to be the hub of all things genealogical (scaling to hundreds of thousands if not millions of users, various languages etc). Yet like any large company or organization, the church has to perform many of the same tasks. Such as supply chain, physical building upkeep, an online presence, HR and IT. And these are all areas where technology can save you larges amounts of time, money and effort. In the words of the CIO technology is used to "decrease administration and increase ministration".

The "keynote" was a talk by the CIO, Joel Dehlin (an ex-Microsoftie). He discussed how the church is keen to adopt technology, and gave the example of having satellite coverage to broadcast it's global bi-annual meetings, called general conference, world wide. The LDS websites receive on the order of 50 million hits per month from 5 million unique viewers. They are currently in the process of revamping many sites, with their new UI design team. They are also adopting technologies such as RSS, and podcasting as well as asking the community what web services could be exposed.

Some of the challenges that the LDS church faces are a little unique (or challenges that only large worldwide companies would face). One example is setting up internet access to meeting houses worldwide. With growth in 3rd world countries, reliable phone lines make this a challenge. Also the members of the church speak 180+ languages. i18n and globalization are huge issues for them. Another issue somewhat similar to startups at the hands of their investors is wise resource management (the church is funded by the tithes of it's members). So adopting open source technology is one way to cut back on IT resources.

Joel's talk was very professional and it looks like he has his hands full. He didn't drink too much Koolaid during his tenure up in Redmond, since he was very positive about open source, complimented Macs heavily, and is pushing a Java dev stack. (Though he suggested at a meetup prior to the talk that OpenOffice is too "technical" to adopt for his users. He went on to clarify that he didn't like mismatches between the screen view and actual printing and that Excel pivot tables are another area that keep them tied to the cash cow of Office). After his talk, there were some 8 or so sessions that were repeated twice. They included streaming video from stake centers (how to allow members to view large regional meetings simultaneously), next-gen genealogical efforts, technology (java web stack), design, development and community. These talks were recorded so the might be podcast at some point. Keep your eyes peeled on YouTube ;)

I attended the Design session led by Tadd Giles (another ex-MSer), director of the newly formed design team. The Church has numerous intranet and external websites, all of which were designed before this team existed. They design team seems pretty strong with prominent designers such as Cameron Moll. Tadd showed some updates to some of the sites and even showed some code, which revealed use of yahoo grids, prototype, and scriptalicious. I was impressed by the talk. Don't be too surprised if LDS sites start looking web 2.0-ish. (Side notes, internal supported browser is IE. All designers are running Macs)

The other talk I attended was the community talk. Led by Tom Welch (old CTO of Linspire) this was sort of a breakthrough moment for the church. Understanding the power of open source and community wisdom, they have opened a technology site, where they are asking for advice and suggestions on how best to interact with their community. Be it open source, exposing web services, or even making some of the church specific apps easier to use. (Visions of genealogical mashups running through my head)

I would've liked to attend the genealogical session and see what the next-gen stuff will be. There is a post on the CIO's blog about the Java stack (which is nice though I prefer other languages... perhaps it will allow them to migrate to jruby. They admit to looking into ruby and python.).

Apparently someone sent the church the Gartner memo that 60% of companies will be using Web 2.0 in the next 2-3 years.

Apps to watch (Beryl)

2007-01-02T00:00:00-07:00

So, I've had a little column called Apps to Watch, on the right hand side of my blog for a while. Those are open source apps that are very useful, disruptors or apps necessary for desktop linux and that many people haven't heard of these. Since I've been a slacker at blogging recently, I figure one theme I can cover is to go over them and discuss or review them. Here's my current list:

And I guess I can even demote apps if I want. So to kick off the new year, I'm removing Kompose and putting Beryl in it's place. (I blogged about Kompose, apple and open source flattery over a year and a half ago. If I were to update the picture I created then, beryl would blow expose out of the water. I also blogged against xgl/compiz when development went in house of Novell. Looks like the community around beryl has shown the Novell did exactly the wrong thing....) (the below is a picture of the mineral beryl)

I've installed 0.1.4 as soon as it was in gentoo and to be frank I haven't noticed much of a difference from the previous version. I used it for two days and then went back to kwin. But I believe Beryl has a lot of potential and can be very useful in both making the desktop more usable and pretty. Even my mac loving brothers were impressed by the Zoom feature and the wobbly windows and cube desktops. Drawbacks for me are:

Snappiness (it degrades with the number of windows on the desktop. Noticeably)
KDE/desktop integration. Granted I haven't installed aquamarine, but the desktops in the pager are not in sync/updated with the beryl desktops (to beryl every desktop is the first one). (I also don't like how it puts all the applications from every window in the taskbar.)
The kompose like functionality isn't as useful as I thought it'd be. Perhaps since I'm a visual person, I know where every app is on each of the four desktops, so it is easier for me to click on the destop and the app then alt-tab through them.

It might seem that I'm negative on Beryl, but really I'm excited about it. And I think having plugins for it should make it all the more useful. One plugin that I'd like is to actually use expose like functionality at all times. Not sure how useful it would be, but I think I might like to try having your main window at 100% and the windows without focus shrinking down in size, using up the remaining real estate. Too bad I don't have more time in the day....

Firefox extension weirdness with Firebug

2006-12-29T00:00:00-07:00

The good and the bad of firefox.

Firefox is cool (though many times I prefer konqueror which can be snappier and take less memory). What makes firefox cooler are the vast extensions (not plugins) available for it. This gives users the chance to have the web how they want it, which can be drastically different that how large (or small) corporations want it to be for you. Adblockplus, flashblock, and del.icio.us complete all enhance my browsing experience. I've been hearing wonderful things about firebug for a while, so I tried installing it, which was pretty simple (on gentoo).

I restarted my browser and the little check in the lower right hand corner was there so I clicked it. The I double clicked it, middle clicked it, and finally right clicked it until a popup box came up. But nothing happened when I selected anything there. Wow what a cool plugin!

The bad is debugging on firefox is a pain.

Perhaps the extension doesn't worked on firefox 2.0.0.1 on gentoo? I searched the bugs and the forums and didn't find anything. Then I opened Tools -> error console and noticed a swath of errors such as:

Error: _CC[cName] has no properties
Source File: chrome://firebug/content/xpcom.js
Line: 58

Error: FBL.ns is not a function
Source File: chrome://firebug/content/sourceCache.js
Line: 36

Error: FBL.ns is not a function
Source File: chrome://firebug/content/insideOutBox.js
Line: 36

...
Error: FBL.initialize is not a function
Source File: chrome://firebug/content/chrome.js
Line: 96

Error: Firebug has no properties
Source File: chrome://browser/content/browser.xul
Line: 1

Error: Firebug has no properties
Source File: chrome://firebug/content/chrome.js
Line: 778

I'm at a loss of where to start at debugging this. But the extension works on my ubuntu box (well, except for the fact that the silly DOM Inspector extension isn't installed by default). I'll post any updates I can find here.

SOLUTION!:Turns out that I needed the "mozdevelop" flag. (Found it by searching for gentoo on the firebug list)

Novell's "Forking" of Open Office

2006-12-06T00:00:00-07:00

This is a response to my friend Matt's take on Novell's work with OpenOffice. I had some trouble posting there, so I'll post it here.

This is definitely a Good Thing(tm) for MS. Switching formats on the new version (Office 2007) is one reason for existing Office users not to upgrade (or a reason to switch over to OOo). So having the "compatible with OOo" card in their hand shows they aren't a monopoly and that they have interoperability with Novell's OOo as well.

I think this is bad for OOo and the Open Document Format. Now MS can push their open format, and we know they certainly have much more $ in the coffers as well as the ability to market (one thing open source is quite lousy at). Now that governments and organizations are starting to adopt ODF, MS can go in and lobby/pay them to adopt openXML, allowing them to keep their cash cow alive and well.

Like Matt, I wouldn't call this a fork either, but it doesn't appear that Ximian/Novell has worked well with the OOo community. Why haven't their changes been rolled back in if they are so good? I don't lurk or follow the developement of OOo too closely. (Note, I really don't want Mono in OOo, but maybe some of their other "features" might be nice). Also, with the new, "we won't sue Novell" relationship, I would be very worried if I were Sun about taking any OOo changes from Novell as this could turn into an even bigger SCO fiasco....

(On the other hand this could be that Sun just doesn't want outsiders developing OOo. That would seem silly, especially how Sun appears to be one of the best behaving Open Source contributors/citizens currently)

If you can't beat them, join... no wait, don't let them license it...

2006-11-23T00:00:00-07:00

Interesting move by MS to license the rights to create fancy new office 2007 UIs. The one exception is that you can't use the ui in a competing product [ read OpenOffice ].

This just seems silly to me, but hey you gotta protect the cash cow somehow...

So what happens when wx, or qt include a ribbon in their toolkit and some project like say koffice just happen to use it? I guess it would be good press for koffice (especially since it should run on windows/macs soon) Any of the 5 koffice people out there listening?

Gentoo, i810, AIGLX, Beryl and white screens

2006-11-03T00:00:00-07:00

So, last night I tried installing Beryl (since AIGLX and Beryl are now in gentoo portage). It seems that the beryl fork of compiz is going well (I wrote about xgl/compiz's (slow) development models a while back).

But even though the community has taken this very promising project and ran with it, there are still some issues. I followed the directions for AIGLX and Beryl, but all I get when I try to run Beryl is a pretty splash screen and then all my windows turn white and grey. I can move them around and see the smooth movement and fancy warping, I can even do the expose like effects. But it is all with white windows.

And the following error messages:

beryl: pixmap 0x4001cb can't be bound to texture
beryl: Couldn't bind redirected window 0x2800007 to texture
beryl: pixmap 0x4001cf can't be bound to texture
beryl: Couldn't bind redirected window 0xc0005a to texture
beryl: pixmap 0x4001cd can't be bound to texture
beryl: Couldn't bind redirected window 0x2000012 to texture
beryl: pixmap 0x4001d1 can't be bound to texture
beryl: Couldn't bind redirected window 0x180000a to texture
beryl: pixmap 0x4001c5 can't be bound to texture

I'm not the first one to come across this, since there is a FAQ entry here (which doesn't say to downgrade xglproto), and a post in the forums, a bug for beryl, and people hitting this outside of gentoo as well. It looks like some have successfully fixed this. Downgrading to gproto 1.4.7 didn't work for me ( emerge -1 -av =glproto-1.4.7 mesa xorg-server xf86-video-i810 beryl-core ).

Hopefully my collection of links above might help others who find themselves in a similar situation. I'll try to update if/when I get it working

Tales of a real life switcher

2006-11-02T00:00:00-07:00

I assume most have seen the Apple switch ads. So being the dork I was, I wondered what a Linux person would look like, if he was invited to the meeting of this odd twosome. Well after two or three years of trying with one person I'm finally going to get to see fruits of my linux labors. Unlike some who espouse the merits of open source, but then go around proclaiming apple to be the solution to the worlds problems, I'd like to hope that linux can work on the desktop as well.

My father has given up on Windows. Or rather, it seems, Windows (98 to be exact) has given up on him. Over the years, my brother or I have had to play IT man to this machine that is now apparently so riddled with spyware/bots/viruses that it barely chugs along. (I guess I should mention that this is a 7 year old 400 Mhz machine). But my dad (like most users) is resistant to change. I gave him my old linux machine (800 Mhz) two years ago, erased my gentoo install and put ubuntu on it (I'm now regretting that decision somewhat), but it just sat there in the basement, functioning fine for basically two years next to this box that should have been placed in some sort of digital leprosy colony. (I figured the linux machine would do everything he wanted, but he showed no interest in it. Plus I was fed up with supporting the windows box...)

Finally yesterday, the straw broke the camels back. There were so many keyloggers on this old windows box that there was a significant delay every time you typed something. My father finally asked me how to print something using the linux box. Turns out he started using it to browse the web.

So to make a long story short this box is using kubuntu on hoary (now I wished I still had my gentoo install, because I'm going to install edgy now...), his printer just worked (hp deskjet 712C). His scanner doesn't work (visioneer 7000?). Openoffice should serve his document needs (word processing and spreadsheets). One more issue, he has been using Microsoft Money to manage some financial data. Hopefully gnucash can help out (I need to convert his .mny files)? I haven't tried gnucash for 6 years, so I can't comment on it, only it seems that it is still a pain to install (even on the latest ubuntu, hence my desire for gentoo's fabulous portage). Lazy web, are there any other recommended solutions other than gnucash?

A few parting comments, my dad asked me, "So will I get viruses?". "No" was my reply. He also mentioned buying a ridiculously overpriced apple box (I guess those ads have affected him too). Seems he would be willing to try Apple, but linux scared him off. I'll try to track any interesting developments in my dad's progress.

Linus on proprietary software (BK)

2006-10-19T00:00:00-06:00

There is an interesting thread in the bazaar-ng mailing list going on right now. Anyway, to make a long story short (and save you from reading ~90 posts in the thread), Linus (of Linux/git fame) said the following concerning BitKeepr which I found quite insightful. (Over a year ago I blogged about my take on the bitkeeper debacle:

On a more personal note: people say that BK showed the "failure" of using a commercial closed-source program. I would disagree. Not only did the kernel get a whole lot of useful work out of BK, we learnt how distributed systems should work, and quite frankly, I'd do it all over again in a heartbeat.

If there was a "failure" in the BK saga, it was in how horrendously bad all open-source SCM's were, even with BK showing how it should have been done for several years. THAT is the failure. The fact that there were hundreds of people who whined about BK, and nobody really did anything productive.

Now, I'm obviously biased, but I really do believe that git is the best open-source SCM there is, by a mile.

Interesting, my take is that those of us who only use free software should leave our bubbles every now and then and copy (I mean be inspired) by the latest and greatest of the proprietary wares of the world.

So everyone running linux, go crank up wine and run that fresh new version of IE and be inspired to deliver on the next web 3.0 applications. (And whine because you miss adblock plus) Seriously I like how Linus can be blunt and pragmatic (without offending too many people).

Alex Martelli and Python 2.5

2006-10-13T00:00:00-06:00

I was able to attend Alex's presentation of Python 2.2 through 2.5 last night. Alex can be a passionate speaker at times, and is very knowledgeable. He went over most of the changes in 2.5 in pretty good detail.

Part of the fun of listening to Alex is when he comes out strong with his opinion. He stated that the with statement was 80% of the reason to use/migrate to 2.5. It seems clear that he wasn't very fond of conditional expressions nor [ctypes])http://docs.python.org/whatsnew/modules.html#SECTION0001410000000000000000). Regarding ctypes, he said that it makes it very easy to crash the python interpreter. Someone in the audience asked if it would get better and he replied something to the effect of "Well, it can't get any worse, at least with C, the compiler will give you some type check warnings".

If you are interested in the changes to the python language in the past 4 years or so and have an hour and a half of time to watch the video check it out when it comes on google video (or youtube?). In the meantime here's the slidedeck.

youtube and google

2006-10-13T00:00:00-06:00

I heard from a friend last night that youtube broke even in the last quarter and was profitable last month. I had always heard that they were shelling out millions for bandwith. Interesting, so they actually had a business plan and were successful at it. The money came from old school banner ads. So the google money making machine should be able to milk that for all it's worth...

The good and bad of python code

2006-10-09T00:00:00-06:00

I recently made a list of things to do to improve ones python coding abilities for a co-worker. One of the suggestions I gave him (besides writing actual python code) was to read other's code (hopefully it's good code). So my question to you python folk is what code would you put on the pedestal for being "pythonic"?

I know of one project that routinely examines Bazaar's codebase to see how they do things. One nice thing about the bazaar mailing list is that they often dissect patches, so it seems that the code quality is pretty good. What are some other stellar projects out there that have great code? (What the code actually does isn't as important as the quality of the code). Any modules in the standard library that stick out? Any particularly good recipes?

Just because python is readable, doesn't mean that all python code is any good. Some common pitfalls of various python programmers appear to be programming as if python were another language (C, Java, etc), not modularizing their code at all (100 line functions if functions at all), no documentation, no testing, or including patterns just to include them.

And for an example of what not to suggest, here's some code <http://files.blog-city.com//files/F05/96843/b/piccopy.py>__ I wrote about 5 years ago. It's sort of the "hello world" of photo management when you are running linux. (A few others have said they have very similar programs). It copies files from one directory to another and in the process creates a hierarchy of date directories to store the image in. (I guess it illustrates, what I consider a wart of python, in that I have to use os, os.path, shutil and stat to manipulate files. As an aside, I'm looking forward to the replacement for the failed PEP 355)

Unwanted marketing (removal with gimp)

2006-10-08T00:00:00-06:00

A few years back we attended a budgeting seminar given by a wealthy Palo Altoan lawyer and his wife. I don't remember much, but I remember his wife saying that she would never buy anything with a prominent logo on it. It was an interesting idea (I admit I kind of at first thought it was an elitist reverse-snob thing), that I hadn't really thought much about before. But basically everything you have/wear with a logo is essentially free advertising for said purveyor of goods.

So when Andrea took this wonderful photo showing off the kids and the fall colors, I had one issue with it: James' hand-me-down sweatshirt. Well with a few minutes using the gimp's color sucker and airbrush tools, I was able to modify it more to my liking.

*Note we actually have quite a few goods from "Oaf", but this glaringly huge logo was just bothering me for some reason.

Mounting remote filesystems using sshfs on ubuntu

2006-09-27T00:00:00-06:00

Here's a quick command line version based on this blog entry, but containing some steps that it doesn't cover. (I've done this twice in the past month or so, so I thought I'd share the joy)

SSHFS is an adaption of FUSE. FUSE, for those who don't know, allows filesystems to be implemented in user space instead of kernel space. So in essence, SSHFS allows you to mount a remote filesystem and access it as if it were local. It comes in handy for me when one of you machines has vpn access, or a source controlled directory and you want to use it on another machine without having to worry or mess with copying and merging. Obviously there is a slight performance hit, since any access to the mounted filesystem has to go over an encrypted line. I should/could probably use samba/nfs for my purposes, but this is a lot simpler than messing with smb.conf (and to be honest I've never looked into setting up nfs).

edit /etc/apt/sources.list and uncomment this line deb http://us.archive.ubuntu.com/ubuntu/ dapper universe
sudo apt-get update
sudo apt-get install sshfs
Create a mount point, here's what I used sudo mkdir /media/r52
Change the owner of the mount point sudo chown matt /media/r52/
Add yourself to the fuse group sudo adduser matt fuse
Log out at this point and re-login so that the group membership takes effect
I mount it like this sshfs -o allow_other r52:/home/matt /media/r52/ (the allow_other option allows other users access to this partition, and you have to add user_allow_other to the file /etc/fuse.conf (make sure there is a newline following it)
If /dev/fuse doesn't exist run this command sudo mknod -m 666 /dev/fuse c 10 229

Mowing the old fashioned way (Brill Luxus 38)

2006-08-31T00:00:00-06:00

I purchased a Brill Luxus 38 from Clean Air Gardening a bit back.

In an age where Home Depots are full of riding lawnmowers, why did I make such a Luddite purchase? There are various reasons listed at the People Powered Machines website. Here's some of mine:

No buying gas
Not polluting
Low price (relative to gas mowers)
Low maintenance

I've mowed a bit of lawns during my teenage years, so I'm not coming into this completely naive. (Note I also ride a geeky recumbent bike around town when able, so I might be a little more green than most)

So far, it has worked pretty good. I've mowed my lawn two times. It takes about 40 minutes. My lawn is pretty small (.25 acre lot) and flat. Turning around is a little tricky, since with a prop mower you are still chopping during that time, but with a reel mower if there is no velocity for the mower, there is no cutting of the lawn. Cutting edges where your grass is taller than say the flower bed below it is a little tricky as well, since you can't mow along the side of the edge or the blade will bind. You have to mow perpendicular to it.

As far as the effort required to use the mower. It requires a little more than the propaganda on the aforementioned websites would suggest. I definitely sweat a little when I mow (and I wouldn't consider myself too out of shape).

All in all, I'm pretty satified with my purchase. Hopefully I'll get many good years of use out of it. The sad thing (much like Open Source btw), is that most people haven't heard of, or haven't considered such mowers, because they are bombarded with miniature SUVs with blades on the bottom, when a simple, cheap, effective solution exists. Is this a marketing problem? Greedy capitalism (No/less money in cheap goods)? Or just catering to lazy couch potatoes?

Dude, where's my umbilical cord? (sis900 ethernet not working)

2006-08-29T00:00:00-06:00

I seem to have no end of networking issues recently.

Last night, Andrea had to reboot an Acer 5003 laptop to do some online banking in Internet Explorer. The machine hibernated. When I rebooted in Linux this morning, the ethernet failed to work.

I rmmod'd the module (sis900) and modprobed it back but no go!

"dmesg" gave the following hint:

eth0: Media Link Off

That churns up lots of hits in Google, the few I read weren't useful, since these people couldn't get the port working at all, and I know that it was working yesterday.

So I rebooted into windows and ethernet was working fine there. I shut windows down completely, (no hibernating), and got the following:

eth0: Media Link On 100mbps full-duplex .

I'm not sure what was going on there, but I'm not going to to touch it...

Hope this helps someone...

The times they are a-changin'

2006-08-28T00:00:00-06:00

Come gather round people.... (yes I've been listening to too much Dylan lately).

So my trusty 4 year old wireless router (netgear wgr614 rev1) apparently gave up some sort of ghost recently. The ethernet ports still route, and the wireless still sends out a signal, but I can't get an IP address via wireless to save my life. (Which I was able to do at some point the in recent past). When I try to connect to it, it just hangs waiting for an address. I'd normally complain about something like this, but since we live in an age where anything electronic wears out faster than a newborn grows out of size 1 diapers, I'm pretty jaded right now.

Luckily Office Depot sent me some sort of $20 off any purchase of $75, so I figured I might as well get a new router. Maybe one of them fancy ones with WPA or something so my neighbors can't steal all my precious data. (Sadly, non of my neighbors have wireless either, so I can't just borrow it from them....)

I ended up purchasing a D-Link, WBR-2310. (I also ended up not getting $20 off my purchases, since the fine print on the back of the coupon said it wouldn't give the discount if they purchase was of "technology"...., talk about a stupid coupon, but I digress). Oh, I said I was jaded before, well, now I'm bonafide jaded, because I also bought one of them 2-year insurance policies for my new router.

So anyway, the point of this post is not to make fun of the silly coupon that caused me to spend $150 on "technology" thinking that I'd get $20 back. The point is that I open up the CD that states "BEGIN HERE" and the first thing I see is an Linux Penguin and an Apple logo and the text: For Mac and Linux users please use the documentation on the CD.... Very cool. Not as cool as if the logos were on the actual box of the device, but cool nontheless.

It looks like normal companies are starting to see that people are using linux! Whoda thunk it?

Sadly, the WBR and all of it's "Rangeboosting" technology is going to be returned very soon. It doesn't play fair with my ActionTec GT701 dsl modem. (I can't even use wireless or the wired switch with that son of a gun).... So maybe I'll just update to a Netgear WGR614 rev7 (or whatever they are at now) (I like the web interface of Netgear much better anyway).

Utah (Salt Lake area) DSL or Cable suggestions....

2006-08-11T00:00:00-06:00

Dear (Utah) Lazyweb-

We are moving into a house in Bountiful, Utah soon. Anyone in the area know if I have any options other than Qwest (DSL) or Comcast (cable) for a high speed connection? Any reccomendations? Are people satisfied with the service they get? If so, who are you using? Quest appears to have a fast (5Mbps) service using MSN as the ISP.

Though there are a host of other ISPs. Is MSN going to require using Windows? (Cause I'm running Linux)

thanks much for any advice or experiences...

Warconfig - feedback requested...

2006-08-04T00:00:00-06:00

I'm working on an open source tool for managing war files called warconfig. We have a need for this at work. I have a pretty detailed write up here. Warconfig is a tool for deploying, upgrading, and rolling back war files on Tomcat (currently only version 5.5 on Gentoo has been tested, though with a little love it should work on Windows and anywhere else that tomcat does. Support for additional app servers should be pretty easy to implement).

The coolest feature, (which is still in the process of being implemented), will allow one to easily see what has changed between war file versions. (Say you have a configuration file containing data specific to your deployment, such as jdbc connection parameters. This will allow you to quickly validate if you've updated your files).

Like the gentoo tools it was based on, it is written in python. This is a quick project and has been developed and tested in the past couple of days (unittests have made my life a lot easier, during various refactorings throughout the week).

So my question to you, lazy web, is what do you think of a tool like this? Is it useful? Overkill? Redundant? How do you manage configuration changes in your J2EE deployments? Please feel free to post your comments below.

The good and bad of customer service (Southwest and Dollar Rent a Car)

2006-08-01T00:00:00-06:00

I'll admit it, I dread customer service. (And I've been on both sides, since I did a brief stint in the service organization at my previous employer). And I've blogged about it before as well as another time. So in the spirit of "Cancel my AOL account", I proceed...

I've been on the phone a bit this morning so I thought I'd vent into the ether of the internet. I'm making a trip to headquarters next week and was originally leaving Sunday afternoon. Well, plans changed and I needed to fly out on Monday morning instead.

So I re-arranged my flight. It was easy, I called Southwest, waited on hold for less than a minute, was greeted by a pleasant person, who charged me $26 to change my flight. Quick and relatively painless. If only all phone calls were the same...

Now the ugly. So I went to Dollars website to change the reservation, below is a screenshot (note that my "modified reservation" is for half a day less than my current reservation).

The gist is they want to charge me twice as much, for picking it up half a day later. So I called their phone reservation system. Sat on hold for a bit before running into a human who insisted that I would have to pay twice as much, there was nothing she could do, but transfer me to customer service. I sat on hold for longer and came accross the same response. Finally I was transferred to the manager of customer service, to whom I told, I was amazed they'd charge me double for driving half a day less. She (obviously couldn't care less about my patronage) and said there was no way I could keep my same reservation and just pick up the car late. So I hinted that I would take my business elsewhere.

She replied "Is there anything else I can do for you?".

"No, you haven't done anything for me."

"Thank you and have a good day."

Click.

Bad memory? Ubuntu (memtest86) to the rescue...

2006-07-28T00:00:00-06:00

My gentoo desktop machine has been acting flakey recently. Apps stop responding, network inconsistent. Sounds like a hardware problem. So I figured I would check my memory.

Memtest86 to the rescue. Thanks to Ubuntu I had a live cd with memtest86 on it. Actually I had 5 of them. So I put one in the machine and let memtest crank away for two days....

Commercial interlude. (No I don't work for Ubuntu.) It is cool that Ubuntu sent me 5 cds for free. And stickers as well. (Though my laptop running ubuntu still has some issues). So use Ubuntu if you aren't willing to run Gentoo)

Back to our (ir-)regularly scheduled program.

Turns out the memory error was at 422.7M (of a 512M module). Good, if it were in the first couple of megs things might be a little more complicated. But I'm happy the error is towards the end of my memory (chances are 50% that it will be). So I added mem=420 to my grub options and I'm happily running now (error free, albeit with 20% less memory). But at least now I can boot to a stable environment if I want to install the badram kernel patches and make my kernel ignore only the bad bits of ram. (Sure wish that patch was included in the gentoo-kernel....)

Missing one image....

2006-07-19T00:00:00-06:00

I saw this a few days ago linked to a blog (apparently here is an older version), if I can find where the original came from I'd give them credit.... (Google, how do I seach on an image, not for an image????) Anyway it was being passed around in email for and I saw a blaring exception. So I spent some time with inkscape. Here's the original:

And here's my addition:

SLLUG Meeting: Setup/Configuration of Django/Lighttp

2006-07-19T00:00:00-06:00

This evening I attended a SLLUG meeting about deploying django with lighhtpd. I've never messed with lightty before, only heard off hand complaints that it doesn't like python too much.

The perl-esque configuration setup seemed to provide for virtual hosting setup pretty easily (much like webapp-config does for gentoo and apache).

Clinton De Young, the presenter, noted that the documentation was very poor for lighhtpd, and that he had to pull some hairs to get everything working. I try to post his configuration when he posts it. Supposedly it will be the only place on the web where it exists (though I'm suprised that ROR doesn't have something you could use...)

(Interesting to note that he had messed around with ROR but gave up on it because lack of unicode support. Didn't like turbogears, since it was less "cohesive". His only complaint was that the documentation for django could be better).

As an aside, I did some django work recently (thanks for fixing my bug), and have been impressed by the community there and the people leading it.

More computers, less keyboards. A plug for synergy2

2006-07-07T00:00:00-06:00

I've been working mainly with two laptops recently. I normally use ergonomic keyboards with my desktops and was noticing that my wrists were sort of twitching. The solution? Buy a new keyboard. (So I went out and bought an ergonomic keyboard that would work with my laptops (read has usb instead of ps/2). I got a MS Ergonomic 4000 and must admit that I like MS hardware. (Own two other ergnomic keyboards and 2 wireless mouses that all come from the evil empire)).

The other annoying thing about switching between laptop keyboards is how variable the key layout is (I'm quite partial to my thinkpad, but the ctrl isn't in the same place as others and once you get used to it, you find that you confuse emacs quite regularly on other keyboards).

The solution to this problem? Synergy2, which lets you use one keyboard and mouse with multiple computers/screens. It is an excellent piece of open source software because:

Easy to install (both ubuntu and gentoo had builds for it, with matching client/server versions)
Excellent documentation. I read halfway through the first page and it was up and going
Easy to run. After you edit a simple configuration file on the "server" (the machine with the keyboard you want to use), run "synergys" on it. Run synergyc SERVER_NAME on the client machine and you should be good to go.
Doesn't bork if one of the computers is pulled off the network
You can copy/paste between computers. This is quite nice. (Note I've only used it with linux thus far, but is has support for windows and mac as well

Take care when emulating IE under Firefox

2006-06-28T00:00:00-06:00

I came accross one of those annoying sites yesterday that refused to serve pages to me because I was using firefox. (Even more annoying since I didn't happen to notice any activex junk). So I told firefox to spoof Internet Explorer. It appeared to work until firefox came crashing down (that's annoying since I tend to have about 20 browsers/tabs going at once). I was running Session Saver which usually helps in cases like these, but upon re-launch firefox immediately crashed again. And continued to do so upon subsequent launches. So I ran firefox from the command line and saw the following:

$ firefox
/usr/libexec/mozilla-launcher: line 339: xdpyinfo: command not found
No running windows found
INTERNAL ERROR on Browser End: Expected a version > 5! Version = 4

System error?:: Success
firefox-bin exited with non-zero status (255)

I googled (using konqueror) for "INTERNAL ERROR on Browser End: Expected a version > 5! Version = 4" and came across a mailing list archive that notes the solution (as well as documents a two year old issue). The solution is to find the pref.js ( find ~/.mozilla |grep prefs.js ) file and remove the line containing::

user_pref("general.useragent.override", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");

I'm now surfing with firefox and all its adblock goodness again. Note: Said website that was crashing firefox also crashed on my in-laws windows machine (when using IE)...

Howto make hoops for goaltimate (diy goaltimate kit)

2006-06-20T00:00:00-06:00

One of my favorite pastimes is playing Ultimate. But sometimes you just can round up enough people to play ultimate. When you don't have enough people or field space for ultimate, goaltimate is a viable alternative. It is played on a smaller field with 4on4 instead of 7 on 7.

But to play goaltimate, one needs a "hoop". So what to do? You can buy one made out of a special shatter-resistant sort of PVC at the goaltimate website. Somewhat nicer are the "dischoop" aluminum, shock-corded poles. I've seen both, but didn't want to shell out the $200+ dollars for either of these solutions. So I made a $26 set.

A goaltimate hoop is 32 feet long. When set up, the two ends should be 18 feet apart, and the center should rise to about 11 feet.

I made my set out of normal PVC (which I've heard has the tendency to shatter at weird times, so I've heard, but yet to experience). To get 32 feet and to have something that is luggable I want 8 pieces at approximately 4 feet (not including overlap). I went down to the local hardware store to look at the 1 inch tubing. You want to get the tubing that has the built-in female end, since that provides about two inches of overlapping. If you use the female-female couplings, you only have about an inch of overlap for both poles (which I've heard can be problematic since you need to bend the hoop quite a bit and can lead to poles popping out).

Try to avoid couplings like these!

One problem was that the variety with the built-in female end only came in 20 foot lengths. I'd need to buy 8 of them and cut them down. At ~$6 per length it would come out to ~$50 (with a LOT of wasted leftover PVC). Hmmmm. I also needed two 36inch rebar segments to hold up the poles (you hammer these 6-8 inches into the grass to hold the poles up).

I decided that $50 was too much to spend and was thinking about using the female-female couplings. But then I went to another local store that had the 20 foot lengths (with the built-in female connecter) and they said they could cut the PVC for me (using a big cross-cut saw) as well as only charge me by the foot. They ended up charging me $26 for the cut PVC and rebar together. (Another hint is that if you need to cut the PVC, use a hacksaw instead of a pipe-cutter, since the pipe- cutter might cause damage to the joint somehow... this is hearsay as well). The individual pieces ended up being 50 inches long (4 feet + 2 inches for overlap). (I don't think that they actually overlap 2 inches, but it is close).

(For the mathematically inclined here's the formula to get the exact length 8a - 7b = 32 (8 pieces of pvc total length of a, minus 7 times the length of the overlapping intersections (b) should come to 32 feet. Plug in b and solve for a. For 50 inch sections, it comes out that the overlap should be 2.5 inches, which is about an inch too long but oh well...)

My rebar came rusted, so I also bought a wire brush and some rust- resistant paint to paint it with. Spent ten minutes brushing it, then paint a few layers on it.

Buy a couple sets of cones and you should be good to go.

Here's a few hints for setting up the field:

Since the base is 18 feet apart, first connect 4 of the poles and lay them down flat. This should be 16 feet. Stick out your foot on one end (my feet are about 1 foot long) and hammer in a piece of rebar. Repeat on the other side. That should be around 18 feet.
Put the rest of the pieces of the pole together. Have one person grab each end of the pole and slide them on the rebar at the same time.

Enjoy your new kit. I played with my family (some of whom had played some recreation ultimate before), all had a good time, and many commented that they liked it better than ultimate.

Chatty customer service

2006-06-15T00:00:00-06:00

I've found myself using chat support twice over the last week. (Which is kind of weird since I helped develop an online chat system a few years back and almost went to work for a online chat provider....) Anyway, one was useful, this one less so (I'm referring to the agents not the chat application).... (Names removed to protect the innocent)...

matt: I need to get a VOIP phone working from home and it requires a static ip address. Any suggestions?

SupportFolk: Please click here for FooCompany DSL technical support via chat or by calling 1-888-777-7777. Chat support is available 10:00 am - 7:00 pm MST, Monday - Saturday. Phone support is available 24 hours a day, 7 days a week.

matt: thanks

SupportFolk: FooCompany appreciates your business and values you as a customer. Our goal is to provide you with excellent service. If you need further assistance, please click here for additional customer service options on FooCompany.com. To close this chat, please click the close button at the top of your chat window.

matt: are you a robot?

matt: ;)

SupportFolk: No (beep beep)

matt: Just checking...

SupportFolk: Is there anything else that I can help you with today?

matt: nope, have a good one.

SupportFolk: You too.

matt: bye

SupportFolk: FooCompany appreciates your business and values you as a customer. Our goal is to provide you with excellent service. If you need further assistance, please click here for additional customer service options on FooCompany.com. To close this chat, please click the close button at the top of your chat window.

Note that this is the online chat help service for the dsl company. One thing always useful with support is the "we can't help you here, but these people can" routing mess that you always seem to end up with....

Googleearth on Linux! Give me back my web

2006-06-13T00:00:00-06:00

Kudos to Google for actually being forward thinking enough to release client software for linux. One problem.... I guess my Thinkpad is a little underpowered for this app. It is way too SLOW! Painful and annoying to use. (Plus the fonts are ugly!) I guess I'm back to google maps.

Too bad there isn't any support or community help. (Yes I have accelerated openGL with my Intel 915 card, is QT part of the same rendering thread? Because even QT refreshes slowly...)

Here's my output at startup:

$ googleearth
libGL warning: 3D driver claims to not support visual 0x23
libGL warning: 3D driver claims to not support visual 0x25
libGL warning: 3D driver claims to not support visual 0x27
libGL warning: 3D driver claims to not support visual 0x29
libGL warning: 3D driver claims to not support visual 0x2b
libGL warning: 3D driver claims to not support visual 0x2d
libGL warning: 3D driver claims to not support visual 0x2f
libGL warning: 3D driver claims to not support visual 0x31
do_wait: drmWaitVBlank returned -1, IRQs don't seem to be working correctly.
Try running with LIBGL_THROTTLE_REFRESH and LIBL_SYNC_REFRESH unset.

update With the help of the Thinkpad Linux mailing list I got the following suggestion, which actually made the thing usable::

export LIBGL_ALWAYS_INDIRECT=true

Now just have to get the fonts working....

My nerdy family

2006-06-13T00:00:00-06:00

My sister started a blog recently. I'm very proud of her (and glad that she isn't using myspace). She's even Web 2.0ey, cause she does things like youtube. I guess I'm just outdated. She has posted a video of my brother John playing with mentos and diet coke.

Perhaps if we get lucky, my brother will pull out his potato gun and we'll get some footage of that.

With my brother we hacked together some 200$ pens and I took him along to a python user group. My wife thought the $2 pen felt better than the upgraded version, though I disagree with her....

On the move! (Moving tips with U-pack ABF)

2006-06-02T00:00:00-06:00

Long time no post. Been busy with a few things. One of them was moving to Utah. We were deliberating on how to move our belongings from California to Utah. Do the Uhaul thing and pull one of our cars? Pay for movers to do everything and just drive accross Nevada in our cars? We ended up using ABF (U-pack), and were pretty pleased with the results.

U-pack is sort of a cross between a Uhaul and a traditional moving compnay. They park an emtpy trailer for you to load. When you're done packing, you put up a retaining wall. They pick up the trailer, fill the remaining space with freight and then drive it for you. That was the appealing part for us and the quote was actually below what it would have cost to rent a Uhaul (note that U-pack included gas, U-haul does not). We didn't really want to drive a Uhaul accross the Sierras. (Too many horror stories of friend's doing the same and having a breakdown). So here's some of my hindsight on moving:

Call U-pack two days before to verify dropoff time
Call U-pack one day before to verify dropoff time
Call U-pack two hours before to verify dropoff time (Note the dropoff time was later than expected by a few hours, which had us a little nervous since we had people committed to come over and help load)
Though pretty, storage containers like this will give way when under they are piled too high.
Try and pack in the same size of boxes for maximum tetris-like cramming of items
Double layer (cardboard) boxes are a lot more crush resistant
Watch craigslist (set alerts) to collect moving boxes for cheap (or free)
Don't make the mistake of packing the mattresses first. Pack them later so you can use them as an additional retaining wall
Try and park on a level surface....

Gems from Startupschool 2006

2006-04-29T00:00:00-06:00

I was able to head up to my alma matter and attend most of Startup School today. Here's my "gems", I tried to note the themes that were common:

Joe Kraus (Jotspot)

Persistance Pays - (See Paul Graham below too)
"No false positives" - Will reject a number of possibly good candidates in order to not hire taint employee pool.
"You make what you measure" - Metrics pay off. The only goals you'll be able to make are those that you measure.
"Better to be a (early) trend spotter than trend setter" - Position Jotspot as a wiki company (because people know what a wiki is), even though Jotspot does much more. Ride the buzz wave. (See Tim Oreilly below)
"Put business model into 'beta' when you put product in 'beta'"
Taking less funding allows for more exits options....

Page Mailliard (WSGR)

Start company off on the right legal footing
Incorporate in Delaware as a C Corporation
Don't start a company with someone else while they are doing a internship at Microsoft. It's possible that MS owns anything they did.
Best leverage for a term sheet (funding from VC) is another term sheet.

Mark Fletcher (Bloglines)

Started 2 companies, ONElist and Bloglines. ONElist raised $48 Mil in funding, Bloglines was self funded with 200K. Recommended no/low funding as possible.

"Release Early/Often" - Product doesn't have to be perfect (See Paul Graham too below)
Listen to users - They will give you good ideas that you never would've thought of. They will be loyal if they see that you are listening and implementing features for them. (See Paul Graham/Caterina Fake here too)
Use Elance - Outsourced a small windows utility for $500, as well as translation and ppt creation.
"Viral growth" - Make it easy for users to tell others about your service
UI - see http://shufflebrain.com/etech06.htm
Hardware - recommended renting machines instead of owning.

3 Ann Winblad (Winblad-Hummer Ventures)

When pitching to VCs need to hook them in 15 minutes
Admit your assumptions. "Know what you don't know"

George Willman (WSGR)

This talk seemed a little out of place, since most of the audience seem to be pretty much against patents. Especially after they were defined as a "right to exclude, not a right to practice".

Tim Oreilly

"Watch the alpha geeks" - These are like the early adopters from crossing the chasm. They can indicate coming trends.
"Find a parade and get in front of it" - Marketing

3 Paul Graham

Supposedly this talk will be part of a forthcoming book.

Hardest Lessons for a startup

Release Early - Makes you work harder since it is released. Can update based on user feedback
Keep Punching out features - Just like muscle building, improvements allow for more improvements. Customers also like improvements, so this also helps with marketing.
Make Users Happy
Fear the right things - Don't fear google (they are now susceptible to innovaters dilemma since they are big (Note Chris Sacca of Google also echoed this when he stated that the worst thing about google is that it is now a "big" company)). Shouldn't worry about google or established players, worry about other startups since they are like you, "cornered animals".But competitors are not the biggest threat. Those are:
- Internal disputes
- Inertia
- Ignoring user (worst)
Commitment is a self-fulfilling prophecy People succeed because they are determined, not smart or intelligent.People are more likely to buy you (finance you) if they know that you will stick around. (Otherwise they will miss out on a good deal and you will improve)
There's always roomFor adding features. We won't be using google search in current state in 10-20 years.There is no limit to the number of startups, since they make wealth.
Don't get your hopes upAssume the worst about the machines and what other people will do. Only can assume the best for yourself (shield your optimism). Assume that deals are not going to happen, then if things work out then you can be pleasantly surprised.

Caterina Fake (Flickr)

Marketing through blogs and viral useage
Online communities need a host, just like any other social gathering.
The Flickr app as known today was one of many apps they were making. This one seemed to get the most traction. They listened to their users (were in the forums for "24x7") and were able to attract more users.

Om Malik (Business 2.0)

Successful apps/companies are those that cause behavior change:

Flickr
Delicious
ipod
"Web 2.0" seems to be catering to the "firefox folks" and ignoring the "ie folks"
SIMPLIFY - Should be understandable very quickly.
Time is the only finite thing. Every thing else is quickly becoming commodity. If it takes 15 minutes to understand it, it's going to fail.

Chris Sacca (Google)

Google now has a bio-diesel bus, with wifi to San Fran. And is the largest single consumer of organic goods in California. They also have a cafeteria that only creates dishes with foods found within a 150 mile radius.
Google has given some ~$80 Million in "founder awards" to internal employees who have created something worthy of being acquired.
All Web2.0 companies are boring, and not doing anything innovative. They all look the same.
Create something that normal people will find useful. (This includes people in 3rd world countries). Oh, by the way, Google donated some machines to a school in India
Google.org is poised to help society more than Google.com
Google doesn't provide breakfast, lunch and dinner to make you stay at work all day (as well as laundry, hair cutting, oil change, etc). They do it because people are creative/innovative during mealtime.
Chris posits that people are only "in the zone" for 1-2 hours a day.

How would you teach python to your co-workers?

2006-04-26T00:00:00-06:00

I've been asked to throw together some impromptu sessions on python for some developers at work. So I'm asking the lazy (python) web what would/have they do/done? My current plan is to be in a lab-type enviroment (everyone will have a laptop/desktop). All attendees are already developers. The first session will probably be just walking through the first chapter of Python Essential Reference, which does a good job of covering the basics. A later session might deal with debugging, coding style, gotchas and testing. Any suggestions, tips or experiences for migrating experienced developers? I've taught 80 year olds and 15 year olds python, but not developers. Hopefully this will prove to be easier ;)

Tips for Python code auditing

2006-04-10T00:00:00-06:00

Tags; python, opensource

Caught this gem lurking on the Portage-dev mailing list. If you are doing a code audit of python code you would do well to follow these tips (courtesty Brian Harring, ex-gentoo dev). Sprinkle that with unittests, pymetrics and pychecker and you could have some fun spring cleaning your code. (Note that Gentoo's python code isn't necessarily "pythonic" or simple. PyMetrics reports the complexity of many of the functions at > 30 (complex/high- risk). Also some of these are gentooisms (ie tabs instead of spaces))

Bootcamp? Where do I sign up?

2006-04-06T00:00:00-06:00

So there's a new "camp". Bootcamp. Come down to Cupertino for a day or two and mix and mingle with a hip crowd of others who want to install web2.0 operating systems on their hardware.... Oh wait, call me when they have XenOfAppleCamp.

Howto create a basic screencast with Linux

2006-03-31T00:00:00-07:00

Learn with me as I experiment with pyvnc2swf. Hopefully I'll have a short series of howtos on creating a screencast (including mixing sound and that sort of fun stuff). Also, this isn't really linux specific per se, but one of the only ways to do it on linux (on mac/windows there are pay for programs that do this, but I believe you can also use pyvnc2swf on those platforms). First you'll need a few things for a basic (soundless) screencast:

vnc
pyvnc2swf

After you have these tools, you are ready to roll. The screencast below should illustrate how to capture content using pyvnc2swf. Enjoy and stay tuned for more. Next step: adding sound with audacity.

JSON-RPC python client code

2006-03-27T00:00:00-07:00

Like Duncan, I've also been messing around with JSON-RPC lately. I'm got most of a WSGI implementation done and a python client library done. I have an article describing the client code here.

For pythoneers, I've got a question: This code is largely based on xmlrpclib. It could easily be implemented as a subclass of xmlrpclib.ServerProxy (instead of just copying the code). So my question is should I subclass it? I'm not doing it right now, since I think it might actually diverge from the xmlrpc code at some point, and it also is a little easier to understand since all the code is in one single file (in my opinion). Opinions? Ideas? Thoughts?

Theory vs practicality

2006-03-23T00:00:00-07:00

I read an interesting article by Gurdy Leet, an art professor, who migrated from closed source tools to open source tools (and operating systems). They moved from MacOSX with AdobeCS to Ubuntu (ppc) with Gimp, inkscape, scribus.

Perhaps just as interesting is the dichotomy found within the comments. People shouting that Leet should be fired by doing a diservice to his students by not preparing them for the industry are at one end and people claiming that his students can focus more on the creative process, than getting bogged up in the tools.

Hmmm, my CS studies felt very similar to what Leet was doing. We focused on the theory. I didn't learn any perl, python, xml, web programming, eclipse, visual studio, etc in school. Most of my programming was done in C (with a little Lisp and Java). In fact my most practical "realworld" class was my senior project (a colloborative browsing engine, using javascript, applets, servlets, perl, apache and mysql). Yet here I am being productive and learning tools after learning the theory.... (I admit that I've met good programmers who just learned tools though, with no/little theory).

Uniden and the disposable phone (TRU-8885-2)

2006-03-15T00:00:00-07:00

My wife or I have recently had the experience of interacting with the customer service folks of 3 different companies. I'd like to share my experiences here for a few reasons:

To draw attention (both good and bad) to the companies treatment of their customer
To allow others to make wise purchasing decisions based.
To see if perhaps the blogging medium can be used to exert (good or bad) pressure on companies to improve their customer service.

In the spirit of full disclosure, no one is paying me to blog this (and I don't work or actually know anyone who works at these companies).

Incident 1 - Gerber

My daughter, Olivia, just celebrated her forth birthday. When she was around six month old we purchased a few Gerber "sippy cups". Note that these cost about $4 for two. They served her well, and I am proud to say that she has graduated from sippy cups and is now using normal "adult" cups. Our son, James, is about 14 months old and for the past 8 months or so has been using the same sippy cups that Olivia used.

For those who aren't familiar with sippy cups, the main selling point is that when you sip, stuff comes out. When you don't sip, nothing should come out. Hence Gerber calls them "spill-proof cups". This modern miracle is achieved by placing a small one-way rubber valve inside the lid. The stopper valve (it's the white thing in the picture above) is actually removable (to enable easy cleaning and is held in place by the friction provided by a small plastic cylinder inside the lid, that the stopper fits in. On one of the many lids we have, the retaining cylinder had broken causing the stopper valve to easily fall out.

My wife, Andrea, decided to call Gerber customer service and see if she could buy some new lids. She claims that she wasn't put on hold for very long, and the lady she talked to was quite sociable. She was asked if the lids were for a boy or a girl (to send us pink or blue lids) and for her address. About 3 days later, 5 new lids arrived. Total cost to us $0. Gerber even paid for shipping. In fact, I'm sure Gerber lost money on this. Or did they?

Let me repeat this. My wife called originally to purchase new lids. She never intended to worry about the warranty or anything like that. These were about 4 year old cups and have definitely seen better days! Andrea and I were pleasantly surprised. If I'm in the baby section and looking to make a purchase I will probably purchase a Gerber product now when given the choice.

Incident 2 - Timbuk2

A month ago I bought an IBM (Lenovo) Thinkpad. I also bought a Timbuk2 Commute Bag to protect my investment. About a week and a half after I bought the laptop friendly messenger bag, the "heavy-duty stitching" on one of the velcro sections started to separate. It wasn't a big deal, about two inches of stitching had separated. Since I knew that Timbuk2 isn't afraid to stand behind their product, I decided to contact their customer service and put them to the test. In a few hours I recieved an email response to send in the bag. A week after sending in the bag, they sent me a notice that I had a $100 voucher towards any product on their website. I'm now awaiting my new bag. Total time cost to me was about 5 minutes on the web, and 5 minutes packing my bag. Since I have to pay tax and shipping (sorta bothers me that I get taxed twice here) on my new bag, plus shipping to send in my defective bag, the replacement is costing me ~$25. Is it worth $25? Probably not for the small tear. Had the problem been bigger, then most definitely. But there is a lot of peace of mind that comes from a company that stands behind their product.

Will I purchase from Timbuk2 again? With a warranty like that, I might not need to. Andrea also has a bag from them that she really likes, and I like the laptop bag. If I'm ever in the market for a new bag and Timbuk2 has one I'll definitely consider it.

Incident 3 - Uniden's disposable phone TRU-8885-2

Less than 4 months after purchasing a Uniden TRU8885-2 phone, it ceased working. After spending more than a half hour waiting for a service representative to speak at her, my wife was told that the phone wasn't under warranty (we had purchased a refurbished model that was warranteed (unknown to me at the time)). So I decided to write to the folks at Uniden:

Dear Uniden-

I'm writing this letter to inform you of my dissatisfaction with a Uniden Phone. I purchased a Uniden TRU8885-2 phone in August 2005. In December of 2005 (less than 4 months!) it stopped working. The base would only display "Welcome. Please wait..." and nothing more. I shipped the phone to be replaced and was informed that "[the unit] was determined to be a refurbished product not certified by Uniden America Corporation. Unfortunately, this is not covered under the manufacturer's warranty.". This is very disturbing and frustrating. Allow me to enumerate why:

Uniden sells consumer electronics. Your brand should be the most important part of your company. Yet I was able to purchase (at Fry's electronics) a phone clearly labeled to be a Uniden phone. There was no indication that the phone was without a warranty. As a consumer of your product, I feel that I have been misled and that your brand name has been misrepresented. Why would you allow your products to be sold without standing behind them?

After the phone stopped working, I began to search the internet and found many other people who have experienced the exact same problem. If this issue is so common, why do you continue to sell this phone? Why would you re-sell refurbished models that have the same problem? Clearly, you are not concerned with costumer satisfaction if this is the case.

My phone cost $90 and it worked for less than 4 months! Did I purchase a disposable phone? I might have expected this, had I purchased a $20 cheapo no-name unit. But I would have expected much more from Uniden.

You insult me by telling me that not only is my phone not under warranty, but that you'll replace the broken unit for $59.95. By the time shipping and handling is added into that, I might as well have purchased a brand new unit.

Furthermore, why would I pay so much to replace this unit? I have no warranty on it and no guarantee that the replacement you send me will last longer than 4 months!

I've been without a cordless phone and answering machine for over a month now.

If you refuse to replace the phone, then in good conscience, you force me to warn others away from your products just as I should have been warned away. To that end I will never buy a Uniden product. I will inform family, friends, coworkers and others via my blog of this situation. Hopefully this would be one of the first hits in Google so that other consumers understand Uniden's stance on product quality and customer service. In other words, I don't want others to make the same mistake I did. Please kindly resolve this matter as soon as possible by sending me a new, reliable phone that will renew my faith in Uniden.

Sincerely, Matt Harrison

As you can tell, I was pretty annoyed with the phone. I recieved a form letter stating that they will replace the base for $60. Andrea has since replaced the phone with a $40 model (non-Uniden). It is currently serving as well or better than the Uniden and we made sure that it has a warrantee.

If you find this post informative, please bookmark it (digg, delicious, furl, stumbleupon, etc), trackback to it, or feel free to email links (or copies of it) or plot it out on gmaps and mash it up. Anyone out in the blogosphere been able to make a difference in customer service with their blog? I'd love it if Uniden got word of my blog again. Help me make it happen.

Pycon2006 Tutorial Recap

2006-02-28T00:00:00-07:00

I attended the tutorials the first day of Pycon. The first one I attended was Python 102. I guess I figured it would be more like 302, so there wasn't much new. Since I'm still basically programming in the 1.6-2.2 era I thought it would be good to have a review of Generators (which lasted about 30 seconds), but oh well. I had time to read the documentation and brush up that way.

The second session was Grig and Titus' "Agile Testing". They wrote and tested (extensively) a web application and were able to achieve 90% code coverage (which is pretty outstanding) and automate the whole process.

An interesting diagram they showed was by Jason Huggins (creator of Selenium), which was called the "Testing pyramid". Just like how programmers eat according to the suggested food pyramid, they should test according to the test pyramid (sadly it seems that in the real world neither happens ;)). They said that they didn't do tdd, but rather ted (test enabled development, ie developing tests alongside or soon after). To automate some of the logic they also used FitNesse, which allows one to create business logic tests easily via a wiki interface. They also used TextTest to do some black box testing.

Using Titus' creation Twill (and WSGI Intercept) they were able to profile and provide coverage reports for common user interactions. This was useful because Twill can automare real user interactions which are more important to profile and cover than random unit tests which may test interactions seldom encountered in real life. (To automate generation of Twill tests, here's some code I wrote).

Finally for testing the actual gui, they used Selenium. All their tests were automated and run using BuildBot following a checkin (or every six hours).

They main walkaway points are that automation of tests is necessary for them to get run (otherwise they will bit rot). Holistic testing will allow one to have more confidence in their tests, and that running tests under a different environment will expose bugs that may get overlooked.

The missing Pycon presentation...

2006-02-28T00:00:00-07:00

This screencast should've been a presentation at Pycon. It does a relatively decent job comparing and constrasting J2EE, ROR, Zope, TG, and Django.

I talked to a handful of people at Pycon who said they would've enjoyed a panel discussion with the web framework creators discussing pros/cons and future areas for collaboration.... (Warning, it's a 400MB download...)

Mashupcamp - Recap

2006-02-21T00:00:00-07:00

Here's my recap of Mashupcamp. My definition of a mashup hasn't really changed since the camp has ended (for some reason I thought that it would). I would define a mashup as a webpage consuming non- native webservices (sprinkle liberally with googlemaps) which may be screenscraped or proxied.

One notable mashup that was missing was greasemonkey! I think that is the epitomy of mashups, yet it wasn't mentioned much at all.

This was an "unconference", so not much was organized beforehand with respect to the sessions. The self organizing aspect of the conference went pretty smoothly. The idea was to get together people who were doing mashups, have some people possibly be creating mashups at the same time, and share experiences. At the end of the camp the best mashups were voted on. Only one of the top 3 used google maps!

Podbop.org won first prize (a Sun Server hand delivered with a Jonathon Swartz appearance, and some quip about mashups finally realizing the 30 year dream of "object reuse"). Adrian Holovarty won 2nd prize (I gave him my vote) with one of the granddaddies of mashups, chicagocrime.org. He won an imac. The 3rd place winner was flyspy, a site that predicts prices for flights that looked pretty nice. All of the top 10 vote winners won a copy of Visual Studio 2005, which was kind of ironic given that most (if not all of them) were LAMP applications.

It's always nice to be around smart energetic people who are doing cool things. You get inspired by what they are doing and inspired by them and their nifty hacks. I'm not so sure I like the "camp" style conference. I've commented on my issues with "camp" previously, but that was more tongue in cheek. The free form structure might be good for foocamp or barcamp that have a wide range of issues to discuss, but the mashupcamp sessions were very limited in scope and they didn't really branch out too much it seemed.

I enjoyed learning about microformats, and Adrian's demo of Chicago Crimes was cool. I also attended a security session, which was similar to other BOFs on security I've been in. The old rehash that there's a lot of insecurity (no real treatment on how to deal with it). One session on "How to make money with open source" really had little to do with open source at all. The firefox session was interesting from a roadmap perspective, and that there were a few microsofties there (who commented that IE 7 isn't really going to add features per se, but focus on being more standards compliant, which is a good thing. They also admitted to developing on firefox because it has better tools that ie!).

As a techie I would have like more hands on sessions like "how to use googlemaps", "how I screen scraped this site" (including code), AJAX showdowns, architectural issues of mashup based sites. The second day had a few hours of "speed geeking". The idea is that everyone who is demoing is given a table and has to demo in 5 minute durations. That went pretty well, but went on a little long. The best part, as is usual for most conferences, was meeting new and interesting people. (Or people that you already knew through the net).

Gentoo on IBM Thinkpad R52....

2006-02-20T00:00:00-07:00

I recently bought a Thinkpad R52. The first thing I did was install gentoo. So far I'm pretty impressed. The R52 doesn't have fancy features, but it has all the basics. I was deliberating whether to wait for the core duo laptops to come out and just getting a laptop now. I figure I won't be running Xen too much, and I don't want to be spending time getting everything working on a first gen laptop. Thinkwiki already has lots of good info, and most of the stuff in my laptop is standard issue stuff. I'll detail my steps I've taken thus far. Later I'll try to push back onto thinkwiki and the gentoo wiki pages, since I don't have everything working right now.... This document is a live document so it is not complete right now..

Gentoo 2005.1 install

The install was a normal gentoo stage 3 install and went quite quickly. (My fasted install yet, but it's also my fastest computer...)

make.conf, USE flags

Here's my current make make.conf... Will fill this in...

Sound

Alsa worked out of the box, I just followed the gentoo instructions and needed to use alsamixer to boost the volume levels.

Udev

I attempted to mess with udev to detect my usb key (which I successfully used on my other gentoo install). The sata drives (which appear as sda) appeared to confuse udev and no device entry was created. I read the gentoo KDE documentation and it now says to use dbus, hal and ivman. Installing those allowed for painless mounting of usb keys and cds in the /media directory. I'll probably go update my desktop with these packages.

x.org

i810 with the stable (6.8) x.org didn't work. I needed to install the modular 7.0 version. That involved lots of editing of package.keywords, package.unmask and package.mask. After the editing I found a script that does that. (I added around 110 packages, not fun!) To get the the 1400x1050 resolution working I needed to add 915resolution package and configure to add the 1400x1050 video mode to the bios. The kind folks from the dri mailing list were prompt to reply to my questions. After upgrading X and installing 915resolution, DRI seemed to fall out and worked (1000fps for glxgears). I have gotten the trackpoint working, the touchpad working and external usb mice to work. This involved editing xorg.conf and udev rules. The synaptics driver will not load for some reason giving the following error:

(**) Option "PalmMinZ" "200"
(EE) Touchpad no synaptics touchpad detected and no repeater device
(EE) Touchpad Unable to query/initialize Synaptics hardware.
(EE) PreInit failed for input device "Touchpad"
(II) UnloadModule: "synaptics"
(**) Option "CoreKeyboard"
(**) Keyboard1: Core Keyboard But the touchpad is currently working with the normal ps2 mouse driver.

Trying to start syndaemon (/etc/init.d/syndaemon start) gives the following::

Can't access shared memory area. SHMConfig disabled? [ !! ]

Apparently I'm missing something. I have attempted to get the external vga port to work on my lcd montitor (though I want to get it working for projectors). My attempt was unsuccessful as I hit Fn-F5 I could only toggle between a black screen and a screen that had two quarter size screens side by side occupying the top half of the monitor.

fonts

The fonts were looking pretty bad even after I got 1400x1050 working. (Well the xterm fonts were crisp...) To rememedy that I just followed the gentoo font documentation and firefox and kde started looking really good.

wireless ipw2200

Wireless took a little more work to get working. This is actually one area where the gentoo documentation is a little lacking. I tried to get things working with wpa_supplicant but it kept crashing. Finally I reverted to wireless tools and was able to get it working. "FOOBAR" is my local ESSID, I have other ones for work. Since my nameserver changes when I'm at work and at home, I had to remove the "nodns" option. "iwconfig" told me that my wireless device was "eth1" , so I symlinked /etc/init.d/net.eth to net.lo. I can now start wireless at work and home by running "/etc/init.d/net.eth1 start". Kwifimanager has proven pretty pointless, and I haven't been able to actually use it to configure or connect my wireless card. The relevant areas of /etc/conf.d/net are:

#wireless wireless-tools
modules=( "iwconfig" )
config_eth1=( "dhcp" )
#dhcp_eth1="nodns nontp nonis"
dhcp_eth1="nontp nonis" #REMOVE THE nodns so /etc/resolv.conf gets updated
essid_eth1="any"
preferred_aps=( "FOOBAR" "any" )

I also added the following to "/etc/modules.autoload.d/kernel-2.6"::

ipw2200 led=1 hwcrypto=0 disable=1

Hibernate, sleeping, acpi etc

Gentoo has pretty good power management instructions and I was able to create a battery profile that steps my cpu speed down when the AC cord is pulled. My first (and only) attempt at suspend to ram failed (I needed to reboot). And I haven't messed around with it more. (The laptop boots pretty fast anyway right now).

Mashupcamp - Screenscraping

2006-02-20T00:00:00-07:00

One of the session's I've attended was by Adrian Holovarty on Screen Scraping. Adrian (of Django fame) made chicago crime to help the citizens of Chicago. Citizen ICAM, police department site that has a database of crimes but is not user friendly for browsing. Has a pretty bad interface only allows 14 days of searching. Embedded at the bottom is "tabular" report, which actually has useful information. Wrote a screen scraper which looks at the latest 90 days, in two week segements. The search radius only allows one mile search radius. Edited the form and changed the radius to 90 miles (and it worked!). Used the live http headers firefox plugin to follow http stream. Adrian pulled all the data from the 90 mile radius. Then next step was to geo code (long/lat). Yahoo maps provides a good geocoding api.

When he originally wrote the website, yahoo's api didn't exists. By scraping google maps (not in terms of use), you could also get this information. Google banned Adrian for this. The basic philosophy is look at Citizen ICANm and do everything the opposite. They don't let you browse, Adrian does, by many different types (street, date, district, zip, location, etc). Make everything in it a link, a permalink (some ajax apps don't let you bookmark).

There is also a route finder, which shows crimes along a user entered route. Using postgis, it allows one to do fuzzy searching for GIS. This allows one to do searches like "see crimes within 2, 4, 8 blocks". It also lets you find your police district or ward. Another journal publishes a police blotter. The police blotter doesn't publish the case number (primary key), so he tries to deduct the case number against the crime database by searching for addresses (by hand). (Yahoo term extractor doesn't find it). Can't post the link to this right now. Chicago PD also has a most wanted list that links photos with case numbers. In the future this will be linked in to show the perpetrators mug shot. A side benefit was that Adrian got to meet with the cops. They called him the day chicago crimes was launched.

OSBC - Nick Carr - OS and the utility revolution

2006-02-20T00:00:00-07:00

I was at OSBC last week, manning the SpikeSource booth and was able to slip into Nick Carr's keynote. Nick Carr basically compared IT today to the state of electicity in factories in the early 1900s. During that time factories had their internal power plants that weren't added value to the outcome of the factory per se (ie not a "core competency"), but were a money sink.

Over the course of 20 years, technology and business saavy allowed people to move from having their internal power generators to take advantage of low cost centralized power generators. Nick said today that IT was a "mess of cables", but in the future IT would be like an electrical socket (ie you just plug in). Buzzwords of the future of IT include: grid, virtualization, SOA, on-demand, blades, etc...

Nick shared a principle that stated the supply of any business resource will gravitate towards it's most economically efficient model. Because IT is a general purpose technology it can take advantage of economies of scale. But this consolidation requires new technologies. Nick then reviewed the history of electrical consolidation in the United States. Basically it was fragmented (self-hosted), a combination of technology (via Thomas Edison) and business (Samuel Insull) allowed the consolidation to take place.

Some of the advances in technology include the emergence of AC, transmission lines, current converters, standardization of networks and AC electric motors. On the business side, Insull invented "modern utility". Insull thought that electricity could be sold as a utility, which apparently was a novel thought at the time. And the factories followed.

Over the next 20 years, the rate of utility suppply roughtly correlated with the inverse of the private supply. Now to apply this to IT. Lots of IT is redundant while many costs are fixed no matter how much they are utilized. He quoted that server, pc and storage utilization rates were 10-35%, 5% and 25-50% respectively. CIO's are currently squeezed. They are spending 70-90% of their budget on infrastructure that is undifferentiated. They would like to spend more on innovation and R&D. Technological advances that could allow for utility IT involved open source (allows for economies of scale and is very flexible), fiber optic networks, IT automation and "Current Converters" (virtualization, grid, web services).

Interesting point Nick made was that although IT was supposed to automate many tasks, IT itself is the last thing to be automated. Carr opined that what is now needed is the new "Insull". A pioneer who scales the IT utility model is lacking. Some mannner to introduce metering and pricing for utility is also required. And the last bullet Carr mentioned (which might be the hardest to tackle) was helping companies overcome the assumption that they should own IT (a huge marketing challenge). It was an interesting analogy. Certainly there are many IT areas where the analogy should hold.

Python in Startups (SDForum)

2006-02-02T00:00:00-07:00

There was another portion of the SDForum meeting titled "Building your startup using Python". It was moderated by the creator of BitTorrent, Braham Cohen and had Steve Kirscho (CEO Propel), Francis Kelly (Founder Loomia), and Peter Yared (CEO ActiveGrid) on the panel. It was interesting to have an assortment of startups that are using python extensively in different areas (client, infrastructure and web service). I've blogged on Activegrid previously, but didn't really know what Propel or Loomia did.

ActiveGrid provides a "commercial LAMP stack". They have an apache mod that lets people create apps that scale easily. They have an application builder created using wxpython, and enable developers to program in python, php and perl using their framework. Since they come from a Java background, they are using lots of XML, SOAP, WSDL etc. (The demo is impressive, I'd call them the anti-thesis of Rails, since ROR tries to avoid XML situps, ActiveGrid appears to be doing situps all day long). Peter mentioned that one of the issues he had with python was the lack of XML tools. When pressed on what that really meant, he said SOAP tools.

Peter came from a smalltalk background before delving off into Java, so he said returning to Python felt natural for him. Most of his developers get up to speed on python quickly and haven't had much trouble grokking it.

Propel provides network "accelerators". They have clients that you download and install on Windows and Mac. According to Steve, one can't tell that they aren't native. Kirsch has had lots of experience using Python and worked with Infoseek (who had a python based search engine). Supposedly Propel's competitors use C for their implementation and Propels python version is faster and more featureful. Kirsch attributes this to python allowing their engineering team to focus on algorithms and not implementation details. In the Alex Martelli session the issue of pythons lack of code compiling (hiding or obfuscation) was raised, Steve briefly mentioned that it was possible to obfuscate python code (probably somewhat important to one delivering proprietary client code it python).

Loomia appears to have a web 2.0 search engine. Francis mentioned that he began to learn python about 2 years ago and it came pretty quickly to him. They are building their services ontop of Zope. Francis said that while python was easy to learn, Zope had a really steep learning curve.

There wasn't much time for questions in this session, but the basic gist of it was something like this. Python allowed these companies to rapidly build and deploy applications. Along they way they ran into hurdles that they might not of had previously. These hurdles might be incomplete documentation, or incomplete implentation, basically "stuff not working". But the notion was that the stuff would start working, either by their own efforts of the efforts of the community behind tools that they were using. And that writing in the missing gaps in python was not too hard.

Braham noted that he re-implemented some of twisted's functionality in bittorrent because twisted didn't provide him what he needed. He also wrote his own SCM system (talk about NIH! :) ), but he also told people he didn't recommend that route.

Python at Google (Greg Stein - SDForum)

2006-02-01T00:00:00-07:00

As I promised, here is the writeup of Greg Stein's keynote last week at the SDForum Python Meeting. All text in italics are my thoughts. This is a repeat of the talk he gave last year at PyCON. He began with his introduction to python and how he began to use it at eShop (after he saw an early demo of what eventually became Zope). Which led into an interested timeline of various companies he had worked for and their attitude towards python.

Attitudes toward Python

1995 - eShop - What is python?
1996 - eShop - Python is great and making us successful
1997 - MS (acquired eShop) - Shipped embedded python in their product
1998 - MS - We are using python? That was a "prototype". Let's re- write it. (The re-write in .asp was slower)
2001 - Collabnet - We don't use python
2003 - Collabnet - Write that in python
2005 - Google - Of course we use python

Greg said that some forward thinking companies are willing to adopt technologies that make them more productive, while some more pragmatic companies will not adopt any new technology unless they see examples of other large companies using it successfully. Me: Interesting that MS is now using python again. Apparently, they have crossed the chasm! (or python has)..

Greg claimed, using python was very advantageous to eShop, in fact it was a "secret sauce". Here are some of the points that allow python to help businesses:

Highly adaptable. One can deal with changing requirements easily. One can also deal with changes in computing environments easily. Both of these can come up in the development lifecycle.
Rapid development.
Easy to maintain.

Google's Engineering Process

At Google, python is one of the 3 "official languages" alongside with C++ and Java. Official here means that Googlers are allowed to deploy these languages to production services. (Internally Google people use many technologies including PHP, C#, Ruby and Perl). Python is well suited to the engineering process at Google. The typical project at Google has a small team (3 people) and a short duration (3 months). After a project is completed developers may move to other projects. Larger projects are sub-divided into 3 month deliverables, and teams get to choose their own language for their project. Engineers are given their 20% time to work on what they want to at Google. Many new ideas spring from this 20% work, and "bottom up" seems to be the mantra at Google. Greg stated that architecture and design were not mandated from the top, but rather the teams working on these projects were given the freedom to suggest and deliver.

Swig is your friend

Google makes extensive use of Swig. Greg indicated that Swig has improved much in recent years. All C++ projects in Google have swig generators created during build time, so python programmers can take advantage of this work. Greg said that neither Boost nor ctypes were as direct or clean as using SWIG.

RPC

RPC is the method Google uses to scale horizontally so well. They have their own internal binary wire format that speaks over http. Programmers can easy speak this format using Java, C++ or Python. Using RPC allows Google to divide computing problems up across large numbers of servers.

Python at Google

Internally Google has been using Python 2.2. It has been hard for them to move forward to 2.3 or 2.4 becaue of the large number of machines that the have and they have to have compatibility amoung those machines ( I'm assumming this is more of an IT issue, since Python is pretty good at backwards compatibility, but I guess if you deploy 2.4 and start using decorators any machine running 2.2 will choke). Greg said that they will soon try to move to 2.4.

Python programmers at Google must follow a strict style guideline (based on PEP8 with 2 spaced indenting). When engineers are first granted commit access to their SCM system, they must pass a style test. All code must pass through two sets of eyes before being checked in. That combined with liberal doses of unittest, pychecker and code coverage eliminates most non-algorithmic issues that might appear in python code.

Where is Python used?

The Google build system is written in python. All of Google's corporate code is checked into a repository and the dependency and building of this code is managed by python. Greg mentioned that to create code.google.com took about 100 lines of python code. But since it has so many dependencies, the build system generated a 3 megabyte makefile for it!
Packaging. Google has an internal packaging format like RPM. These packages are created using python.
Binary Data Pusher. This is the area where Alex Martelli is working, on optimizing pushing bits between thousands of servers
Production servers. All monitoring, restarting and data collection functionality is done with python
Reporting. Logs are analyzed and reports are generated using Python.
A few services including code.google.com and google groups. Most other front ends are in C++ (google.com) and Java (gmail). All web services are built on top of a highly optimizing http server wrapped with SWIG.

Open Sourced Code

Greg commented that the code Google has released as Open Source thus far is not too interesting. But hopefully that will change in the future. He noted that they will probably release their packaging system.

Performance of Python Code

Greg said the really Python is rarely a bottleneck at Google. (With bits going over a network and hitting a database, both of these will impact before python even comes into the picture). As mentioned previosly when MS ported their eShops code to .asp, because "Python is interpreted and has to be slow" (not a direct quote, but something like that). When the port to asp was done, the code was in fact slower!

When programming in python, one should design for scalability. If you design properly, the you should be able to just throw more machines at the problem and your app should scale fine.

Greg's final point on performance was that you can always write in C/C++ and wrap if you need to. This is an attempt to justify the use of python to higher up types who may disapprove of it's use. Greg stated (and this is a direct quote), "People have been saying [wrap C/C++] for 10 years. I've never done this once!"

Questions/Observations

Alex Martelli commented that you can throw more machines to solve the bandwidth problem, but you can't do that to solve the latency problem.

Someone asked about the use of MySQL in Google. Greg said, "We use it specifically because it is open source". When you buy from a vendor you are subject to the whims of that vendor. Google had an instance where they were using proprietary software and needed a feature to be added. The vendor said no. Google offered to pay for a developer. The vendor said no! Google obviously wasn't happy with this, and open source is one solution. ( Obviously not having to pay Oracle a per cpu license for 1000's of machines is probably another reason, but not one that Greg mentioned).

There was talk about testing and code quality of python. Greg said that by using code coverage for an interpreted language. Once you have coverage on a line you know it will "compile" much like a C or Java program. So you can get around typos somewhat with that.

Someone asked why Google even used Java at all? Greg appeared to bite his tongue and then said that there are a lot of good java programmers out there and Google hires a few of them.

On the issue of catching bugs, Greg mentioned that because they are running code on tens of thousands of machines, that they see bugs that appear less frequently A LOT more than most people. They've run into obscure kernel bugs that other people rarely run into. So he said, Google's production code is quite good, because bugs get exposed early and often.

There was a question on the GIL (global interpreter lock) in python. Greg said he tried to remove the GIL a few years back, but it was a quick hack and slowed some things down. He said that trying to remove it now would prove very difficult ( hopefully PyPy helps here). But Greg's suggestion was to use RPC instead of threading.

All in all it was a very good talk, providing interesting insight from a man who has a lot of experience using Python. It was also fun to see where Google is using python inside.

Python web framework shootout, take 3. Petitioning Guido (Google) to choose their framework

2006-01-27T00:00:00-07:00

A year ago I heard Guido say that he wasn't much interested in web frameworks, but apparently "after years of resistance" he's had a change of heart. Sounds like he's going through what every python programmer who wants to deliver web apps goes through. I like the fact that he want's web server independence. That is something that will benefit all python web programmers. (3 cheers for WSGI and Paste).

A few more of my thoughts reading this post and the responses:

Google hasn't standardized on any python framework (though python is one of 3 official google languages)
"Knowing myself, I'd happily go off and build my own web framework at this point". Is that the first thing that pops into any python programmers head whenever they run accross something? Hence the abundance of choices for web programming. I guess we can blame Guido for making it too easy to build stuff in python ;)
Guido is pragmatic. He is looking at the competition, ie ROR.
Guido is more pragmatic. He looks at more python choices before going off and writing his own.
Guido is very pragmatic. He asks the community for their recommendations, realizing that many of these people have given this a lot more thought than he has.
Lots of framework developers want Guido (or Google) to use their framework.

Note this is all good stuff. As someone who is also currently evaluating web programming frameworks, I find this very useful. I guess my pie in the sky dream is wsgi compliance. Then everyone can have their cake, and I can eat all of it. (Or at least if you wrote something for twisted, but I'm using cherrypy, or webpy or myghty or quixote, or zope or django or turbogears or ... or ... or ... I can still use your stuff too).

Note Guido actually commented on this

Why criticize Django for claiming to be the best? Nobody has denied it, and yet it's criticized as if it were somehow unethical. Frankly, the problem is that there are too many Python web frameworks and wannabees, and if we don't start some kind of shootout, however subjective, we'll never get to market dominance of a few good ones. I'm not saying Django is already the winner -- but we are looking for a winner (or, more likely, a small set of winners) so future developers looking for a Pythonic solution only have to compare a small number of options, all mature, feature-rich, well-supported etc., rather than having to sift through 80+ half-baked solutions.

Python in the Enterprise - Guido, Alex and more (SDForum)

2006-01-26T00:00:00-07:00

I attended the SDForum Dynamic Language Series on Python this morning. I'll jot down a few of the highlights here.

Guido and the history of python

The program started off with a history of python from Guido. He had two weeks of free time and wanted to implement a language. He said his main design goals were interactiveness, work well with other tools, avoid NIH (borrow freely), and doable as a one person job. The last one was a little interesting since he explicitly said he would have to cut corners to make it a one person job (seems like this is common amoung new open source projects since the normally have only one dev). He also said that the object oriented nature wasn't due to a deep theoretical understanding of object oriented principles but that it grew out of a need to enable extensibility in the language.

Python in the Enterprise

This was a panel discussion called "Python in the Enterprise" by Munwar Shariff (CTO Cignex, a Zope/Plone shop) and Alex Martelli. The moderator (who was an editor for Software Development, a programming magazine) said she noted that python programmers net about $10K/year more than most programmers. Alex attributed that to being more productive with python.

There was a statement that python's biggest problem was that it didn't have a compiler and hence the source was available. Alex said people can decompile anything, so the ONLY way around this is to provide a web service if you really don't want your IP in the wild. Munwar stated that he had sold product to many government agencies (but his product is also a web service).

There was the standard lack of static type checking complaint. The reply was Eckels quote that code that isn't tested is worthless anyway (I'm paraphrasing a little).

I asked what the panelists thought when I recently asked a friend what was the difference between two products (one in java and one in python), and he (he happens to sell the java version) replied "It's written in python, enough said", implying that no one would buy the python version by virtue of the language it was written in. I don't think Alex understood what I was trying to get at, which was the ingrained notion that enterprise software has to be written in java (perhaps it's analagous to "no one ever got fired for buying from IBM", choosing Java software is a safe choice because you don't have to justify it). Alex said, "Don't tell them what language it is written in." He also mentioned that you could use jython too. Munwar admitted that they had lost a sale because they were pushing the zope platform.

There was talk was actual research that Python was "better", rather than anecdotal evidence. Alex mentioned something report by Franheufer Institute? (I'll have to see if I can locate it). Location is here Munwar offered his anecdotal evidence that developers were 5-10x faster in python, but that performance in python suffered a tiny bit. Stay tuned, I'll try to post a followup with Greg Stein's "Python at Google" and some notes from the "Python in startups" panel soon.

reStructuredText is cool

2006-01-22T00:00:00-07:00

A few weeks ago I blogged about what digital format to use for a journal that would eventually be printed at some indefinite point in the future.

I was deliberating between odt and html. OpenOffice gave me essentially what I wanted, but I thought using a text file instead of a binary file might help out since I'm going to be adding lots of pictures (would rather have a pointer to them than embed them), and I can also do a diff of a text file.

Well, a kind reader recommended I look into reStructuredText and I'm glad he reminded me of it. (I've been meaning to check it out). I should add that I was thinking about using wiki markup as well (I would say wiki is to html what python is to java). But I think rst is actually a very good match for me. Here's what I like about rst:

I can edit in any text editor
The markup is really simple. Here's a template I made for myself:

Jan 22, 2006

Paragraph number 1.

Paragraph number 2.

.. figure:: /path/to/picture :width: 600

Caption for picture

(The really nice thing here is the figure markup. It adds a bunch of div tags around the image and the caption, so if I want to do anything fancy with css it would be simple. I don't know how easy this is in wiki...). + I can open/import it in OpenOffice after I convert it to html (which consists of doing the following: rst2html.py journal.rst > test.html + I can export as tex if I feel the desire + I can spell check in emacs + The file size of my journal will stay relatively small. (I'm keeping backups of my pictures as well and the links should be easy to replace, since I have a straightforward numbering scheme for storing digital pics on my computer) + I can use it for creating an s5 slideshow for pycon...

Downsides? Well, I have to postprocess the file, before being able to preview in html.... Oh well, I like it so far.

Mmmm, CherryPaste with Eggs...

2006-01-20T00:00:00-07:00

I've been messing around with some really fun python stuff recently for some stuff I'm doing at work and in preparation for my Pycon talk.

I think I'm finally beginning to understand Paste, and since I like CherryPy, I thought I'd try out Ian's new CherryPaste. And just to have more fun, I thought I through in some eggs too. The result of my learning is an article that takes the basic CherryPy tutorial code, deploys it using CherryPaste, and they deploys it as an egg. Hope you enjoy your sticky, eggy pie!

What format to use for a (dead tree) journal? (2006 New years goal)

2006-01-08T00:00:00-07:00

I've been blogging for almost a year now and have found it quite enjoyable. Writing is quite therapeutic, and it makes you think about stuff a little more than you normally would. One of my most valuable possessions is a 300 page tome, my grandfather printed a few years before his death (he actually learned to use a computer at age 75 explicitly for this purpose!). Since my memory is about half as good as his (he could remember how much the boat tickets from San Fran to Tahiti cost some 70 years ago), I figure I better get a head start. So my goal for 2006 is to create a personal (non-online) journal. I want to try and give it equal time as my online technical journal. (Since most of my family members don't grok my blog. Though interestingly enough, I've convinced one of my four siblings to start a blog.) I figure my children will probably benefit a lot more from my personal journal, than my forays into programming, technology and Dick Proenneke. So my question to you my technical friends is what format to use? I've actually already started using ODF (you know the format the Massachusetts adopted), but am thinking about maybe switching to html. The end goal of course is a dead tree hard cover tome. I'm using a computer rather than typing it by hand because I want to be able to include digital photos in the journal. Perhaps very simple xml. My main concern is that by embedding pictures into the document, the size of the journal could get quite big, quite fast (hence the desire to link to the pictures via html/xml). Here are my requirements:

Spell checking
Easily editable/readable by myself (or my wife)
Somewhat futureproof

I figure if I use simple html/xml instead of ODF, writing a script to convert it to latex, odt, whatever the latest and greatest format is of the time, I can do it. Viewing diffs from SVN would be easy. Let me explain what I mean by futureproof. If I don't do it in an open format, it might cause other people to have to re-digitize it and hence duplicate effort). I've recieved many family histories that are 3 generations xeroxed copies that started out digital. I'd much prefer to recieve a cd containing the document with the copy, at the very least to preserve the photo quality. (I can re-type the text if necessary, but I'm not sure I can get another copy of some of these old photos). My grandpa's journal is written in word perfect. For me to have it in digital form, I need to find the zip disk containing a copy of it, get that file on my computer and pray that OpenOffice WP import works ok.... (If done in an open format, that last step should be less trouble/worrysome) Eventually, I'll want to print a hard copy format. At that time, I'll do index generation, page numbering, title page, etc. I'm interested in hearing any experiences or suggestions.

The real benefits of Open Office

2006-01-01T00:00:00-07:00

My buddy Matt has an interesting post on the problems with OpenOffice. I'm sad that Matt misses what I feel is the most important contribution of Open Office.

I would argue that Open Office along with Firefox are two of the most important and successful Open Source applications for the mass of humanity (I'm not discounting Linux (which is probably orders of magnitude more important) or GCC(which is important to developers)). I'm grateful to have Open Office and I use it almost everyday. Is it the epitome of good software? No, but it does its job.

Why is Open Office successful? Because it is free in the libre sense? I don't think this is the case. As Matt and other's have pointed out 90 of 100 developers who are working on OO are paid by companies to do so. You might say cool 90 people. I say, why does it take 90 people? (I had the same feeling at the recent FOSS.in conference in the groupware session where I heard that there are around 20 developers working on Evolution, but 1 working on Kontact! And from the demos given, I'm thinking about moving to Kontact).

Is it successful because it is free in the gratis sense? Probably to a degree. Lots of companies would like to lower their licensing costs. Using Open Office is one way to do that.

Is it successful because it is cross platform? Again, probably to a degree. This is important but it is not the main factor.

I claim the main factor for its success is "compatibility". Because we have been entrenched with a monopoly on productivity software, people need to be compatible with that. Granted, it's not 100% compatible, but for 90% of people it does the job. If it wasn't compatible to some degree, few would use it. If it was more compatible with the incumbent format, more people would use it.

Which brings me to my main point. What is the most beneficial thing OO has done? Opening up millions of lines of source code that few people can understand? Nope. Giving away software for free? Nope. What is it then? The most important thing OO has done has nothing to do with the fact that it is open source software. In fact, this thing happened after OO already had compatibility with MS Office. The most important thing is a new standard now supported in OpenOffice2. The ODF standard. This standard is open to the world and anyone can implement it.

Matt complains about Linux World forcing him to use Open Office, but what it appears to me is that they are saying use a standard compliant format. Is Massachusetts betting on Open Office because it is open source? I think they are more interested in a standard. (An quick aside, I was talking with a friend who works for a company that makes movies during lunchtime ultimate pickup today. I asked him if they use homegrown or proprietary software. His response is enlightening. "Mostly homegrown, you don't know if these small companies selling software will be around in 5 years." But if they are supporting a standard, you don't have to worry about being able to access your data in 5 years).

What would happen if you browser didn't render HTML, it only rendered a proprietary version of a markup language that was undocumented? Would people complain then? In fact, where would the internet be without standards?

Looking to the future.... So is Open Office the answer? For most people I think it does the job. Is it the best solution? Probably not. But now that the standard is out there and others are supporting it, expect meaner leaner office environments. My bet is that KOffice will be the Firefox of the office suites (when it uses QT4 which will allow free versions for windows). It will be lightweight fast, and people will be able to extend the code. (How many people are working on KWord right now? 3 or 4?) And it will work on Macs as well, without requiring you to fuss with X.

.Net, Mono and slow compiled code

2005-12-29T00:00:00-07:00

Miguel has an interesting post discussing the merits of C#. This was spurred by Nitesh's email with Steve Jobs regarding the geriatric state of objective c. (While the /. commentary was somewhat interesting regarding this exchange, my thought was "How many CEOs spend their Dec 25th emailing random people asking 1 sentence questions?"... I guess no one else thought that.)

Interesting that Miguel basically dismisses JIT compilation (admitting that handcoded assembly will always be faster). How does that bode for pypy? (And the rumor that Guido is working on it at Google... I asked Guido about PyPy a month ago at BayPiggies and he said something to the effect of "If anyone can make it work, Armin can. I'm not really following it")

But more interesting is that a lot of his arguments for using Mono, could be easily applied to python. But why would you want to write python instead of C# and cut the number of lines in half? Or lose the compilation step?... That surely wouldn't make us more productive...

Examples of movement in the commercial/open source box model

2005-12-20T00:00:00-07:00

Just saw this post about xGL development closing up. My thought from that post was Novell thinks in house development is better or they don't want to collaborate with others (who are interested in xgl, ie users/developers from redhat, debian, gentoo, ubuntu, ...).

So I thought I'd put my box back up here and populate it with the xgl instance as well as other examples. I've put some examples of projects that appear to be going in the opposite direction. For those that are moving to open source, there appears to be commercial activity building on top of them (the curvy line). (Though I'm not sure that the curvy line actually goes to the right, it might go back to the left depending on a few factors). I'd love to hear comments or feedback...

Another picture from India

2005-12-15T00:00:00-07:00

This is the Bangalore Palace. Site of FOSS 2005.

Visualizing the commercialization of open source

2005-12-15T00:00:00-07:00

This morning I had a few hours to myself in a jetlagged induced stupor. It was refreshing to sit and watch the sunrise, yet disappointing to realize I had driven to work only to be locked out for an hour. So I had some time to ponder.

Being the visual person I am, I think that a picture can help us understand (or simplify) things. (Or over- generalize them). So while I was sitting there thinking, I thought of the relationship between Commercialization and Open Source. Without further ado, here's my creation: I think I'll leave this post at that.

Later I'll post some more arrows along with ideas. Thoughts? Make sense? Do we see any trends? Feel free to share your comments. (I can upload the svg file if people want to "fix" it...)

Back from India

2005-12-12T00:00:00-07:00

I'm back from my trip to India. I had a great time. If I don't have anything useful to post about, I'll post pictures from my trip. I'll start out with this gem. In case you can't tell, I'm bowling. No, not that kind of bowling, the other kind, the cricket kind... Our office went out for a lunch time game.

Python web framework rewrite take two....

2005-12-07T00:00:00-07:00

Is the ease of developing in python a drawback? Is too much choice just confusing?

I'm in the middle of trying to get TurboGears working with Python 2.3 (the latest subversion head supposedly supports it...) to see if I can use it as a framework for SAM. (Though like Titus, I'll probably just use CherryPy).

But it is suprising to hear of two more meta/mega web frameworks today: Pylons and web.py. So I guess python just makes it too easy to create your own framework... subway, django, TG...

Interesting that of the above mentioned frameworks they all have some overlap of their sub-components (Django being the notable exception that re-wrote everything).

Also interesting that ROR's outspoken creator, David, says if Django had existed a while ago he probably would've used it instead of creating his own framework.

For my needs I want to be able to expose web pages (and ajax endpoints) with a clear api, rather than using the php-like style in my latest prototype code. Cherrypy should suit me fine. And I can plug in whatever templating engine suits me (Cheetah, kidd, myghty, .... ).

As these mature, will there be consolidation, which tends to happen as things commoditize? Will there continue to be more frameworks created as new people come to python and decide to scratch their itches to build websites?

Generate twill scripts and mechanize unittests from TestGen4Web

2005-12-05T00:00:00-07:00

UPDATE (Jan 2007)Duncan McGrego has taken these scripts and run with them. There is an updated version on the cheeseshop! Thanks Duncan. (Behold open source!) end update

While attending FOSS.IN, one of the sessions was a website security session. I only caught the tail end, but it felt very similar to a session at OSCON this year by Nitesh. I also attended part of Rasmus' talk where he talked about a tool he uses for detecting XSS. Which in turn motivated (reminded) me to examine some of the python tools for programming and simulating web browsers. I checked out both Mechanize and Twill by writing scripts that convert TestGen4Web output to a python unittest and twill script respectively.

Both twill and mechanize seem to work pretty well and appear to be actively developed. I had some issues with twill saying "& was unepected at this time" (still haven't resolved that, though repeating the command doesn't cause a repeat of the error...) while running scripts. Twill appears to nicely ignore robots.txt (and possibly update the user agent?), whereas I needed to explicitly tell mechanize to both ignore robots.txt (as well as change the user agent to impersonate moz). That behavior seems reasonable since twill is meant to be interactive.

One area where both are (understandably) lacking is javascript support. With ajax becoming more popular and widespread, this functionality is likely needed. Though this is probably a case where a pure python solution would seem silly. Python wrappers to the moz object model seem the way to go (don't know if that's possible though).

I impersonate Alax Cox at FOSS

2005-12-05T00:00:00-07:00

One of the nice things about conferences is the oppurtunity to meet new people. It was nice to be able to mingle with Alan Cox for a little bit while fighting for Pizza at FOSS. Talking with Seemant was cool too. Props to those community volunteers such as Seemant who dedicate a large portion of their time without financial renumeration.

I need to comment on one funny thing that happened on the last day of the conference. I was helping out in our booth, when an employee from a neighboring booth asked me for help. This company was quizzing participants of FOSS and giving away copies of "The Mythical Man Month" to high scorers. They had Alan Cox signing and giving out the books (no, Alan did not write that book). Apparently Alan had to take off and they still had a book to deliver so they asked me if I could stand in Alan's place. I also got to autograph my name (right below Alan's), and there's a picture of me on someone's camera delivering a copy of the book to a high scorer. I guess I should start working on kernel development....

FOSS Day 3

2005-12-01T00:00:00-07:00

Today started off with a bang with a keynote Andrew Cowie. He did an excellent job speaking and wasn't afraid to ruffle feathers. One of the best parts was his proposal to change Linux to "Open Office/Mozilla/Apache/Perl/Gnu/Linux" (talking RMS's GNU/Linux fixation one step further). He also questioned the lack of sponsorship by Open Source companies such as RedHat and Novell. "Why do they not sponser community conferences?" I wondered the same thing at OSCON. He also opined why he choose not to use Mono (microsoft embrance and extend conspiracy theory). It was refreshing to hear someone who wasn't afraid to say his opinion when it may clash a little with the party line.

Andrew then discussed how Open Source empowers the developer. The developer controls Open Source because "no one can tell you [not to develop features]". Therein lies the "freedom" in free software. It was pretty motivational as he discussed how he went from being a lurker to a maintainer of the Java/Gnome bindings.

I then attended a series of Groupware sessions. Namely on Kolab, Evolution and Kontact. Till Adams, the main dev for Kolab and Kontact spoke. He noted that groupware was one of the final pieces needed for migration to complete open source systems. One point he said (that I'm not sure I agree with since it kind of flies in the face of the power of the community (though I admit and have blogged that there are some things that Open Source doesn't do well)) was that there are not enought community resources to compete with outlook/exchange. 8-10 developers can not compete with the 1000 or so working for microsoft on these projects.

Till also mentioned that in order to allow people to at least think about migration you need a full groupware stack. Just having a server or client alone, wouldn't cut it. He then described how the German government employed his company for six weeks (yes 6), to write a replacement for Outlook/Exchange. That hacked out Kolab 1 (codenamed Mission Impossible). Amazingly enough (it took about 3 months he said), it worked enough that it was deployed. The agile manner of fixing bugs or adding features really pleased the Government. But, a re-write was needed, one that actually involved the community rather than 3 developers writing code for 24x7 for 3 months. That rewrite turned into Kolab2.

On the client end evolution and kontact are both advancing in features. I was very impressed by what Kontact has in KDE 3.5. (I think I'll migrate from evolution for a while and try it). It's a testament to kde, qt or something that 1 person working in his spare time can create such a feature rich client. (How many people are working full time on evolution?) Till noted that he had worked with OpenUsability.org to make Kontact more usable. From his demo, it appears their collaboration has paid off. One other thing that I found nice was that both Till and the evolution developers are adopting a project called opensync (rather than writing/maintaining their own syncing framework). That's excellent, it's cool to hear that projects can collaborate, rather than re-invent their own wheels.

I also attended part of a talk by Neeti Agarwal, who has been directing some of Novell's open source developement operations and he discussed some of the difficulties in open source and traditional software development culture and Indian culture. He indicated that the open source idea of meritocracy was in opposition to the Indian culture of respect for elders (and seniority). I (not being Indian) found that interesting. He said that hiring for open source was difficult because you basically had to hire fresh grads who liked open source, since normal (non-programmers) wouldn't respect open source programmers. Re-training experienced people to "unlearn" the standard best practices, usually was a lesson in frustration. A big "no no". Interesting.

Another interesting point he said was that most Indians don't care about the debates and whatnot that go on in the larger open source communities (he gave planet.gnome as an example) (he admitted though that the actual process of debate helped to create the leaders of the community). But they do care about local issues. His idea was that there is a need for local leaders, and the way to attract them was to focus on local issues that local communities could revolve around and actually care about. By creating smaller focused communities, he felt that open source could attract more developers.

I attended the tail end of a cross site scripting session (aparently one of the banks in India has a hole) which led to a BOF session on the lack of security in wireless, and people not trusting greasemonkey, javascript and windows. Seemed like some good stuff mixed in with some FUD.

One more session I caught was Danese's presentation on giving "Foss presentations". Her main points seemed to be, don't take yourself too seriously, don't have too many slides, and don't give away the reasons for attending your talk before your talk (don't give them the carrot just dangle it in front)....

FOSS Day 2 (the kernel, turbogears, google and gentoo)

2005-11-30T00:00:00-07:00

Today was the second day of FOSS. I'll jot down my observations from today.

The first talk I attended was from LWN's Jonathon Corbet. He discussed recent happenings in the Linux kernel. He noted that the development cycle of the kernel was becoming more "professional", with two week cycles for adding features (and four weeks for fixing bugs). Jonathon was very well versed in the new developments. It was refreshing to sit back and here some semi- detailed updates on the kernel. I haven't updated mine since the 2.6.10 days.

Later I attending a basic introduction to TurboGears given by Swaroop. It was pretty well attended. I haven't tried to do much with TurboGears, but this presentation made it feel very similar to the Rails demos I've seen (that's a compliment). I like that the TG developers are taking advantage of existing projects (and not re-inventing the wheel). My biggest issue with TG is the Python 2.4 requirement. How many distros are shipping 2.4 these days? (Gentoo and ubuntu??)

Speaking of Ruby and Python, there appears to be a bit of interest by the conference attendees in these languages. Many people were asking about them and what distringuishes them or discussing their use of them.

After some booth work I caught the tail end of "Google: Powered by Linux". According to the speaker, Arvind Jain, Google is one of "the biggest hardware producers". He said they actually make their own machines. Look out Dell you are about to be disrupted by the Gbox! (They'll probably give them away for free as long as they can monitor the keystrokes ;)). Their machines also have a failure rate of 1 in 1000. (For every 1000 computers 1/day will fail). But they don't care since the Google File System provides them with redundancies. Next followed some discussion of their grid abstraction layer (that allows programmers to easily develop grid applications using GFS, GWS (Global Work System) and Map Reduce). Supposedly these tools make it dead simple to create very scalable apps. (Perhaps ActiveGrid can provide the common folk with similar functionality, since Google's stuff is proprietary and only used by Google).

Arvind also mentioned something that they will be introducing in the future called "Big Table". This is their homegrown distributed database. That's about all the details he gave. (No it doesn't use flat files).

Other interesting google snippets: In a two week period of Aug 2004, there were 26,000 worker deaths (so I guess you can estimate the size of their cluster(s) by that). Also, people have forgotten how to spell since 10% of queries are mispelled.

The last talks I attended were the end of Ebuild 101 and using Gentoo in Production. The takeaways here, are that there is a lot of power in portage (and ebuild scripts) that is pretty easy to access and that yes, people actually do use Gentoo in production systems. BUT, unless one knows what they are doing it is "not recommended". One the other hand if you do know what you are doing, it can make administration very simple (especially for large numbers of boxes) with no downtime. I'm very impressed by Gentoo. I think it is a shining example of open source well executed (even the documentation is superb).

Report from FOSS 2005 (day 1)

2005-11-29T00:00:00-07:00

I made it to India intact. I haven't really slept much in the past 48 hours, so forgive the typos. The FOSS conference is being held outside of the palace of Bangalore. The schedule got kind of messed up and many of the talks were delayed by varying amounts. This made it hard to attend some of the talks since some talks ended up overlapping quite a bit.

I caught the end of Rasmus Lerdofs discussion on Cross Site Scripting. He described a PHP tool to find XSS which he created call Scanmus. The tool is not open sourced because he didn't want it to fall into the hands of wrong doers. He used the tool to find holes in the FOSS website and an Indian Airline.

According to Rasmus, website security is broken (on the server side). He compared it to a firewall. Normally a firewall blocks out everything and you have to punch holes into it to open up necessary ports. He argued the server side programming languages should treat input data in a like manner. (All data should be filtered by default). The "input_get" commands in PHP will do this for you, but you need to apply patches for php version 4.

While he was fielding questions I asked Rasmus, if he felt open source (server side) software was more secure than proprietary. I love loaded questions. I think he wanted to say that they are both bad, but ended up saying that he stopped examining proprietary websites since instead of thanking him for pointing out a flaw in their code, they sent the lawyers after him

Next I attended a talk about the Linux Test Automation Framework. One interesting thing about it is that it is written in Bash.

After that I did some booth duty and ate some pizza. In the afternoon, I couldn't find any talks that I wanted to attend (I'd like to blame it on the talks being off by 45 minutes or more, but it's more likely that my lack of sleep was really catching up on me).

The SpikeSource booth was pretty busy. After a while I went and talked to some other exhibitors. Accross the hallway from us was the Gentoo booth. I was able to talk with Seemant Kulleen a little bit about Gentoo and it's progress. He's a very down to earth, pragmatic guy. I half jokingly asked him what Gentoo was going to do to prevent mass adoption of Ubuntu and he flatout said that he doesn't care. He's not in it for the competition, he's in it for fun. He also said flat out that Gentoo is focused on the developer. He also kind of reprimanded me, since I admitted to being a user and "lurker" since 2001 who also can program in python. I told him that at least I file bugs with Gentoo (that a step closer than just being a casual user).

Well, I think my brain is really shutting down on me now. Good night or morning...

Off to India

2005-11-23T00:00:00-07:00

Still alive here. What's new? I'm off to India for a while. I'm going to work with our team over there and help out with Foss. This will be my first time on the other side of the Atlantic so I'm pretty excited.

SAM release with AJAX!

2005-10-29T00:00:00-06:00

I've updated SAM to now include a webserver. This brings along with it a new AJAXy interface (I guess this is sort of a followup to my AJAX post, Alex (dojo) and Bob (mochikit) were the only open source people who responded to my requests for comments (one commercial company started the interview process but ended up not responding...). I'll probably end up using both in the future, though right now I'm only using MochiKit. I think this architecture is pretty cool (and useful/necessary in SAM's case).

Instead of adding a thick client to SAM (which has some drawbacks since SAM will happily run from the command line on Windows, Linux and Macs) which would increase the download by a significant amount as well as increase complexity, and in some cases (headless machines) not even work, I'm doing something a little different.

This first developmental release (0.30) starts up a prototyped webserver (hacked out using WebStack) when passed in the "--ajax" parameter. It will also start up a browser and point it to "localhost:8080/index". The initial interface allows you to choose the components to run and see the results.

This is all pure python right now (well, AJAX is compliments of MochiKit (javascript)), but I could envision other applications using this sort of architecture. For example, anything that one stores on a thumb drive could probably be adapted to use this sort of architecture. That way even though you run on different machines (Mac, Windows or Linux) you can still access the data in the same way (provided you have python installed on those machines (the only one where this is probably an issue is Windows and it's trivial to use py2exe in those cases and create a small ~2Meg exe)) as well as remotely if needed.

For people who want to see a simple example of JSON/Ajax/MochiKit in action feel free to download the .30 release. This architecture should be really straightforward for people who want to understand and look under the hood of the server side of AJAX.

I'm not suggesting using this server side code in any production environment. Basically I took WebStack and created a simple endpoint that dispatches everything through it (I'm pretty sure this is the wrong way to do it, but it took about 150 lines for the whole server, which includes the ajax endpoints).

I need to look at Twisted (a good reason to buy the book), or possibly stripped down versions of Django or TurboGears. I want to keep everything in pure python and as small as possible, but I'd like to re-use others code/best practices when possible. The end goal is the .40 release(SAM is adopting a even is stable odd is development numbering scheme).

Hopefully one will be able to have SAM servers that can discover others (possibly through pyzeroconf) and allow IT folks to control multiple instances from a single browser. Anyway, as you can tell, I think this could turn out to be pretty cool. Feel free to provide feedback (if you think I'm just jumping on the AJAX bandwagon) or if you like the mini-local-client-server-ajax-gui architecture.

TestGen4Web, record/script your interactions with Firefox

2005-10-26T00:00:00-06:00

A co-worker of mine has created a very useful firefox extension for capturing interactions in Firefox. By itself this is useful for scripting common things you do with the web that are tedious yet you do them often. (I had an old coworker who had carpal tunnels who used a similar tool on ie to script anything that he did more than once...).

But then there is the testing functionality. You can save/load scripts as well as replay them in the browser. The scripts are serialized as xml and the project also includes translators to convert these scripts to httpunit (java) and simpletest (php) so they can be run in an automated fashion. (An adapter for selenium could probably be written too...). Perhaps if I get some time (read motivation), I'll try to write an adapter for twill.

The release is in alpha and requires recent builds of firefox (1.5), but is already useful in it's current state.

Google hates me....

2005-10-25T00:00:00-06:00

I seem to have this love hate relationship with the company that seems to control the web. Ok this post will probably seem quite vain, so I apologize for that. But it illustrates that even though a companies' motto is "Do no evil", it doesn't always mean that they are providing the best results. Sometimes the results are just wierd.

Here's a general example. Say you need to find out if certain hardware supports linux. Usually your search involves specific hardware models and "linux" and usually there is a lot of cruft in the results (aka ads and vendors (big brother I know you are reading this, please put that in Froogle where it belongs)).

Here's a specific example. A few months ago my blog was removed from the indexes of google for containing the word "Google bomb", here's the post that explains what happened when I noticed that a previous post somehow floated to the front page of google in some countries. Now when I mean removed, I don't mean removed from the first one or two pages of results (it should not have been there anyway for such a trivial post). I mean, according to google my blog didn't exist. There were NO references to it. A week or two later (after an email) I was allowed to exist again.

Later on I noticed that my vanity search was listing my blog in the top 5 results on google (referrer logs help here). That was pretty interesting and I'm not saying that I'm one of the most important 5 Matt Harrisons, but for some reason I was up there. Then about two months ago, I disappeared again. (I wish I would have paid more attention, so I knew what triggered it.) No, I was still in the index this time, but pushed off into the nether world of ~150 (ie I might as well not be in the index). Why did this happen? I don't know.

Now to end my rant. Call me a complainer, call me vain, but explain to me why my vanity search brings up the Spike Asset Manager (which I wrote, yet has NO mention of me on the page!) as number two! I asked a professional SEO and he can't tell me either. For what it's worth both yahoo and msn list me in the top 10. Maybe I should switch search engines.

How to create divx videos for Philips 642 DVD Player using Linux?

2005-10-20T00:00:00-06:00

Yes that title is a question not an answer. A short amount of scouring the internet turned up nothing. We recently purchased a Canon S1 camera that takes pretty good mjpeg videos. Our DVD player, the Philips 642 will play divx video, yet I can't seem to convert the mjpeg videos correctly. If anyone has any clues feel free to pipe in. In the meantime here's a script to spider a directory and create a shell script to convert all videos found to SVCD (since my player will play those). It also renames the files to include their date in the name.

#!/usr/bin/python
import sys
import os
import stat
import time

class DirectoryWalker:
    """from http://aspn.activestate.com/ASPN/Mail/Message/541112"""
    def __init__(self, directory):
        self.stack = [directory]
        self.files = []
        self.index = 0

    def __getitem__(self, index):
        while 1:
            try:
                file = self.files[self.index]
                self.index = self.index + 1
            except IndexError:
                # pop next directory from stack.  this raises
                # an IndexError if there's nothing more to do
                self.directory = self.stack.pop()
                if os.access(self.directory, os.R_OK):
                    try:
                        #windows might not really have "read" permissions
                        #see http://mail.python.org/pipermail/python-bugs-list/2004-August/024493.html
                        self.files = os.listdir(self.directory)
                        self.index = 0
                    except Exception, e:
                        #print e
                        continue
            else:
                # got a filename
                if file.startswith("."):
                    #skip hidden files
                    continue
                fullname = os.path.join(self.directory, file)
                if os.path.isdir(fullname) and not os.path.islink(fullname):
                    # it's a directory; save it for later
                    self.stack.append(fullname)
                return fullname

class Date:
    def __init__(self, secs):
        self.gtime = time.localtime(secs)
    def getYear(self):
        return str(self.gtime[0])
    def getMonth(self):
        months = {1:"January",2:"February",3:"March",4:"April",
          5:"May",6:"June",7:"July",8:"August",9:"September",
          10:"October",11:"November",12:"December"}
        return str(months[self.gtime[1]])

    def getPaddedMonth(self):
        return self.pad(self.gtime[1])+self.getMonth()

    def pad(self, num, totalWidth=2):
        """pad upto total with on left side with zeros
        """
        zeros = "0"*(totalWidth-len(str(num)))
        return zeros+str(num)

    def getDay(self):
        day = str(self.gtime[2])
        #pad with leading zeros
        return self.pad(day)

    def strftime(self, fmt):
        return time.strftime(fmt, self.gtime)

    def getMDY(self):
        return self.getMonth()+self.getDay()+self.getYear()

def cpDir(indir, outdir):
    for filename in DirectoryWalker(indir):
        if filename.endswith("avi"):
            filestat = os.stat(filename)
            mtime = filestat[stat.ST_MTIME]
            gtime = time.gmtime(mtime)
            date = Date(mtime)
            basename = os.path.basename(filename)
            outname = date.strftime("%y%m%d") + basename[:basename.index(".avi")] + ".mpg"
            outpath = os.path.join(outdir, outname)
            #change this command as desired
            print """ffmpeg -i %s -target ntsc-svcd -s 640x480 %s"""%(filename,outpath)

if __name__=="__main__":
    #crawl arg1 and write a command to output it as SVCD in arg2
    cpDir(sys.argv[1], sys.argv[2])

Fun with technical support and web standards

2005-10-14T00:00:00-06:00

Here's an email exchange I recently had:

Hello- I saw the link to your service today on Slashdot. I'm interested in your photo books, yet I don't have a Windows machine. Are you planning to support firefox anytime soon. If so, you will probably have a new customer. thanks matt

Here's the response:

Dear Matt, Thank you for contacting us. Please note that Mozilla Firefox is totally not compatible to use for [Name removed to protect guilty] site. You must use Netscape Navigator or Internet Explorer Versions 5.0 and above. AOL users should be using the most current version of the AOL software. If you are using earlier versions of these browsers, you are likely to encounter problems. Thank you for your suggestion, it will be forwarded to our product development and research group. We will let you know through our newsletter as soon as it is updated. Thanks again for your feedback. We really appreciate your willingness to share ideas. Please let us know if we can be of any further assistance. Sincerely, Charlie

To which I responded:

Charlie- Thanks but I don't think you even read my email. I don't have a windows machine! AOL and Internet Explorer are NOT options for me. By the way I get the same error with Netscape that I got with Firefox. "We're sorry flipbooks are not available for Netscape users". Well I guess this netscape user will go elsewhere.... Matt

I got back the following:

Dear Matt, We are very sorry for the error in my earlier email to you. We request you to use Internet Explorer for ordering photo books from our site. Thanks for your suggestion. We review customer feedback regularly to determine what [name removed] customers most want and your comments will certainly be a part of that process. Once again we regret any inconvenience you have been caused. Do write in to us, in case of any further assistance required. We will be glad to assist you. Sincerely, Charlie

After that I gave up and stopped wasting my time. Anyone else had similar experiences trying to get online services working on non- internet explorer enabled machines?

MochiKit DOM helper script

2005-10-14T00:00:00-06:00

So, I've been trying out MochiKit. The stuff I've done so far seems very straightforward and also functional (as in lispy). Hopefully I'll have something to show for it soon. In the meantime, I've created a python script (read hack complete with unit tests) for converting well formed html to MochiKit's DOM. I've written an article describing the script on SpikeSource's developer website.

Inkscape and Xara (an example of competion)

2005-10-13T00:00:00-06:00

The recently open sourced (dual licensed ala mySQL) Xara project is interesting. Their FAQ is very frank and straightforward. I like it. The program itself appears to be a combination of vector and bitmap manipulation utilities.

One aspect that I find interesting is that the Xara folks aparently were in contact with the Inkscape folks and discussed their pros and cons of open sourcing the tool.

That thread is pretty interesting for many reasons (the mozilla sidenote included). Also in that same thread is illustrated various opinions of competition in open source. Some note that there could a possible symbiotic relationship between Inkscape and Xara by using a common renderer (which is very fast in Xara). Others note that:

"Competition is tough. There's only so many Linux people interested in vector graphics, and they will need to decide which project to contribute to. Few if any people will be able to learn both codebases to contribute to both. This will hurt us (developers drain) and this will hurt Xara ("hey, we went open source, why so few contributors?")."

Then there's opinions all the way on the opposite spectrum:

"Overall, I can't help feeling very positive about this whole thing. I love it that their motivation - to create the best vector editor on Earth - is exactly the same as mine. And no matter which of our two programs gobbles the other, we now have significantly better chances of reaching that goal than we did yesterday. As for increased competition for Inkscape, I agree with Mental that this is healthy."

It will be interesting to watch and see what happens here. Since it is an example of open source competition playing out before our eyes. Another interesting thing to note is Xara's use of wxWidgets. Will that hinder linux adoption since neither of the two popular toolkit's use it (yes I know wx is a wrapper around gtk, but how many popular wx apps are there?)? On the flip side having the app available on Mac and Windows as well should be a boon. The two big "successful" open source projects, firefox and openoffice, both have cross platform apps. (Note: I'm not saying I don't like wx, I've used wxPython for a bit and found it quite pleasant, I think it's the best choice for Windows Python app development.)

Ajax explosion

2005-10-06T00:00:00-06:00

With OpenOffice 3.0 (the AJAX edition) in the works, I thought I'd join the bandwagon and do a little more with AJAX. So I have a dilemma. Much like in the python world (where it is really easy to implement what you want) where there exists many slightly different flavors of wheels (with new ones comming out too), there also seems to be a plethora of AJAX frameworks. And it appears that new ones are reported pretty frequently.

I've messed around with CPAINT (in fact I ported version 1 to python available here with the AJAX spellchecker), and SAJAX. But I was blown away during OSCON by Alex Russell's Dojo (he seemed to know his javascript inside out, and I found things like graceful degradation and js packaging quite innovative). (Ok, prototype is cool too and was featured at OSCON).

But then there's mochikit, which seems to be pythonic, documented and tested (all good things). Plus it's being adopted by at least one fast moving project (or meta-project ;)). And I've heard good things about open rico too.

So, how to decide? The friendly folks over at OSAF have the beginnings of a comparison. I'm currently split between Dojo and Mochikit. If anyone has used these or others and has strong opinions please feel free to comment here (or point me to where google has not). I'm looking for anything more substantive than the OSAF comparison. Thanks.

My (mini) interview with MochiKit creator (Bob Ippolito) posted with his permission:

I would like to solicit your opinion on what makes MochiKit the library of choice.

B: Well obviously I'm biased here, because I wrote it. I wouldn't have written it if the other libraries were suitable for the kind of code I was writing.

Ok, so the other libraries weren't suitable because....? Because of the code? Because of the app? I'm not trying to be annoying, I think it's interesting to see why people "re-invent" the wheel. Why not contribute to say dojo or openrico so that they are suitable to what you were doing?

B: Dojo didn't exist in a usable form, I don't think open rico hadn't been released yet. Neither have very complete documentation, and it was easier to write my own than to read someone else's source and figure out how to use it. Prototype did ugly things that aren't compatible with (most) other JavaScript libraries and had no documentation or tests, so I threw that right out the window andeverything that depends on it.

If I were starting today, I'd probably contribute to Dojo. I'm not interested in dropping MochiKit for Dojo because MochiKit does everything I need and Dojo doesn't. I talk with those guys prettyoften, and some of MochiKit will probably end up in Dojo at some point... but I'm not the one that's going to be doing that work.

MochiKit doesn't try to do everything (like Dojo), but the functionality it has is very good. I encourage people to use other libraries with MochiKit if you want functionality that is available elsewhere.

Could you elaborate a little bit more about what it does and doesn't do? Why did you include/not include the functionality in MochiKit?

B: I've talked a bit about that on my blog. The documentation covers what it does and doesn't do; MochiKit's public API is completelydocumented.

My mini-interview with Alex Russell (of dojo fame):

What makes Dojo the library of choice?

A: I'm going to cop out on this one and mention that you don't have to choose between Mochi and Dojo. You can use them together. In fact, BobIppolito has added Dojo package system hooks into Mochi so that you can load (and build) Mochi via Dojo.

That said, Dojo has some situations in which it makes more sense than other options. Namely, if you're looking to develop UI components thatwill be re-used in an application and don't want to do DOM manipulation by hand, Dojo is the best tool for the job. Past that, Dojo provideshigh-level abstractions for all kinds of stuff that an Ajax-style app might want to do, including an event system that makes the differencebetween DOM Events and normal function calls dissapear and an IO API that makes the whole "Ajax" bit of ajax much easier. The animationlibrary is pretty great too.

And did I mention that we have a WYSIWYG editor component? And it works on Safari.

Past that, it sounds like you're familiar with the advantages of things like the package system, etc., so I won't bore you with those.

What docs we have are at:

http://dojotoolkit.org/docs/

What do you provide that others (openrico/prototype) don't?

A: It's kind of hard to even remember all the things that distinguish us. The back-button fix, the event system, and the widget system spring tomind though.

OpenRico is based on prototype, as is scriptaculous. They both take the core stuff provided by Prototype and extend it in some way. Neither of them have a coherent widget system (although scriptaculous is trending that way), and neither are designed to degrade or integrate with markup the way Dojo is. Perhaps Dojo's defining feature today is that we are markup-driven both in how you write widgets and in how you declarethem.

Instead of trying to glue a DnD system onto an animation library and then try to combine it with some other widget or "Ajax" system, you canjust pick them all up in one place (Dojo), and safely pick only what you need. And we come with components like combo-boxes and rich- texteditors out of the box.

Want a WYSIWYG editor? it looks like this in dojo:

<!--somewhere in head-->

<script src="path/to/dojo.js"></script>

<script>dojo.require("dojo.widget.Editor");</script>

<!-- ... somewhere in body-->

<div class="dojo-Editor">

+ ...

</div>

And when your document loads, the
will get replaced with a Dojo Editor control (you could also speicify "dojo-RichText" for an editorwithout chrome).

Where all the other toolkits fall down, and where we've invested significant effort, is in deploy-time optimization. Our compressorsystem is seamlessly integrated with our build and package tools. Most of these other tools make it an "all or nothing" affair, and if you want to build a single deployment file, you'll have to call it a different way. The Dojo package system make it transparent. You just build, point to the new build, and it all Just Works (TM).

What is the history of Dojo?

A:Dojo started as me contacting all of my friends from the DHTML community (I did netWindows before Dojo...I've been at this a long time) and basically said "if this stuff is gonna fly, we need a single toolkit. Lets stop re-inventing the wheel". We started a mailing list, got aname, and got down to coding. That was about 18 months ago.

At the time, there were several larger toolkits that people donated, namely Burstlib, netWindows, and f(m), and we set about to pick just the best pieces from each and make a coherent whole.

When you started working on it, did other projects exist?

A:Not the projects you mention, no.

If so, why did you build your own?

That is, perhaps, a better question for all the other toolkit authors. I'm pretty sure that I've approached most of them about cooperation,but many have demurred for various reasons. We started as a consolidation project and would love to continue that consolidation should some of these fragmented efforts get tired of maintaining their own stuff and re-inventing what everyone else has already done.

The Dojo comitters list reads like a "who's who" list of DHTML experts.

Script to find rank in the internet

2005-09-30T00:00:00-06:00

Because I'm fascinated by how your ranking in certain search engines can fluctuate so rapidly over even one day, I hacked out a script (easier than scanning through pages of results). Pass the script a query (in quotes) and the url (can just be the first unique characters of it) you want to know the rank of, and it will tell you the rank. Yes, this is a hack. Yes there are other tools that do this. Yes, it requires Beautiful Soup (which was really straightforward to use).

Here's an example:

$ python find.py proenneke http://panela
http://www.google.com/search?q=proenneke&start=0
found! :http://panela
8

The effect of blogging on other internet communities

2005-09-29T00:00:00-06:00

Just found an interesting post on the RSD (rec.sport.disc, mainly discussing ultimate) newsgroup. It asks if the glory days of the newsgroup are over since many of the characters in the community have begun to blog. Which leads to a few questions:

Will newsgroups whither away?
Will they only be an area to ask newbie type questions?
What happens to soapbox one is given by posting to a newsgroup? (By this I mean that if I post to RSD, I know that a few hundred people (interested in my post) will read my post. How can I now direct a question or comment to this same community if the newsgroup dissolves?)
Will aggregators become the new newsgroup? (Perhaps answering my question above) There is already one for ultimate. But the barrier to entry (and having an audience) is a lot higher than just posting to a newsgroup...

New release of SAM (Spike Asset Manager)

2005-09-27T00:00:00-06:00

An update of a project I work on, SAM (Spike Asset Manager) has been released. The major focus of the recent 0.21 release has been to up the number of components it looks for. It now finds 274 common Open Source components.

From the homepage:

Spike Asset Manager (SAM) is an open-source cross-platform framework (written in python) for probing a system (unix or windows) for components and reporting them. It includes a driver file that probes for components commonly found in a LAMPJ stack (Apache, MySQL, PHP, Tomcat, etc). SAM can find multiple versions that are installed, query the rpm database and indicate whether components are running. Driver files may be edited to search for different components.

The Amazing Real Matt and his dissappearing desktop

2005-09-27T00:00:00-06:00

I actually don't read many blogs directly. One of the few that I read often is my buddy Matt Asay's (aka Fake Matt).

He's been pontificating for a while over there that the desktop is dead, and the only relevant operating system is the internet. And I guess my recent response caused him to clarify his positioning (while posing as a ghost writer).

My argument is that AJAX, while useful, doesn't provide a wholesale replacement for desktop applications. (Interestingly enough we switch positioning over email clients, me favoring gmail, him favoring fat clients (so they can sync to his handhelds)).

I'm a developer who uses the web a lot, but spend a lot of time scripting and debugging. When I'm using a computer at home I'm doing more multimedia type activities (video editing, image manipulation) that can't be replaced by a fancy web app. Matt's coming from a business side.

As Open Source Evangelist for Novell, he spends his time meeting with new startups (and thinking about how best to use open source). For him, the communication mechanisms are important. A lot of time thick clients just get in the way. That's fine with me if they go to the web (I'm all for wikis instead of word processors). And all the better for the googles, and linuxes (while worse for the apples and ms's).

So if the desktop is dying, then I'll mourn its passing the day I buy a laptop. (Not a web enabled cell phone). I don't think I can be productive without some number crunching power on my end.

Using the iPod Nano on linux...

2005-09-26T00:00:00-06:00

So I fell prey to Apple's marketing, and I bought a Nano for my wife's birthday. It's for her, I promise! (Sadly, the 2 day fedex shipping, turned into almost two weeks. It was stuck in China having a "shipment exception" for over 5 days.

Apple customer service claimed they didn't know what the holdup was, just that they were sorry. To be fair they refunded my shipping expenditures, but even discounting the 5 days in China, the "two day" shipping took 4 days (not counting a full weekend)!). Since we have no Windows machine (or Mac for that matter) I'll document my work with Linux. I'm using gentoo, perhaps some of the "desktop friendly" distros make the udev stuff less of a hassle.

Charge the nano. Plug it into a powered usb port.
Set up udev. This can be a little tricky. For more details check out here. From dmesg, I noticed that the Nano was being detected as a usb storage device.:

usb-storage: device found at 2 usb-storage: waiting for device to settle before scanning Vendor: Apple Model: iPod Rev: 1.62 Type: Direct-Access ANSI SCSI revision: 00SCSI device sdb: 7999488 512-byte hdwr sectors (4096 MB)sdb: assuming Write Enabledsdb: assuming drive cache: write through SCSI device sdb: 7999488 512-byte hdwr sectors (4096 MB)sdb: assuming Write Enabledsdb: assuming drive cache: write through sdb: sdb1 sdb2Attached scsi removable disk sdb at scsi1, channel 0, id 0, lun 0Attached scsi generic sg1 at scsi1, channel 0, id 0, lun 0, type 0
Use the udevinfo command to find out about the usb device (don't start looking at /sys/bus/scsi cause it won't work! Ask me how I know)

matt@mycomputer ~ $ udevinfo -a -p /sys/bus/usb/devices/2-2/

udevinfo starts with the device the node belongs to and then walks up thedevice chain, to print for every device found, all possibly useful attributesin the udev key format.Only attributes within one device section may be used together in one rule,to match the device for which the node will be created. looking at class device '/sys/bus/usb/devices/2-2': SUBSYSTEM=="unknown" SYSFS{bConfigurationValue}=="1" SYSFS{bDeviceClass}=="00" SYSFS{bDeviceProtocol}=="00" SYSFS{bDeviceSubClass}=="00" SYSFS{bMaxPower}=="500mA" SYSFS{bNumConfigurations}=="2" SYSFS{bNumInterfaces}==" 1" SYSFS{bcdDevice}=="0001" SYSFS{bmAttributes}=="c0" SYSFS{configuration}=="" SYSFS{detach_state}=="0" SYSFS{devnum}=="13" SYSFS{idProduct}=="120a" SYSFS{idVendor}=="05ac" SYSFS{manufacturer}=="Apple" SYSFS{maxchild}=="0" SYSFS{product}=="iPod" SYSFS{serial}=="000A270012314B59" SYSFS{speed}=="12" SYSFS{version}==" 2.00" Add the following line to /etc/udev/rules.d/10-local.rules (note the spaces). The SYMLINK part will create a /dev/ipod entry for you. BUS="usb", SYSFS{model}="iPod " NAME="%k", SYMLINK="ipod"
Create mount directory

mkdir /mnt/ipod
Update /etc/fstab. Most linux ipod-friendly apps expect a /mnt/ipod. Update /etc/fstab accordingly

/dev/ipod /mnt/ipod auto nodev,nosuid,user,rw,noauto 0 0
Make sure you have the correct kernel modules load hfs kernel module

modprobe hfs
Mount that puppy

mount /mnt/ipod

launch gtkpod
create directories (File->Create Ipod Directories). Necessary the first time.
Put some tunes on it
Make sure you 'umount' before unplugging...

It appears that amarok (v1.3.2 works too). Still need to figure out how to get album covers working (perhaps an amarok script might be needed).

Open Source Mountain Biking

2005-09-19T00:00:00-06:00

This past weekend I was able to go on a very nice bike ride at Soquel Demonstration Forest. Braille trail is quite nice, and something that makes it pretty unique among most mountain biking trails is that this trail evolves. The half dozen times I've ridden it, it has been a little different each time.

While nature certainly has it's way with the trail, there is a also community of people who change the trail to their liking. As such this is also one of the rare trails I've seen that has man made obstacles (teeter totters, ramps, jumps, etc) as well (many obstacles looked like they required large tools to create). It seems that both these changes go hand in hand. A tree may fall over causing an obstruction in the trail. On most trails when something like this occurs, you hop off your bike and portage around the obstacle (until someone chainsaws it clear, or removes the tree if it is small enough).

On Saturday we ran into a similar situation. A downed tree had left an area of the trail unrideable. So we decided to remove the various branches and that the tree took down with it. After that we were left with a 8 inch log going right through the path. But instead of removing this log, we left it in as a feature of the trail, by creating a log ramp up one side and down the other. Only time will tell if the other "committers" of the trail approved of our change. (There was a nice flyer from more ROMP on our car when we finished the ride, advocating responsible trail maintainance).

Using the Samsung ML-2010 laser printer with Linux

2005-08-31T00:00:00-06:00

Our 5 year old inkjet printer has been making loud annoying noises and refusing to print midway through a page. So this last weekend I decided to replace it with a laser printer (we only really used color for printing photos, and since 1hour kiosks are common and cheap at many places and I prefer those to inkjet). (Before you ask, no this isn't a rehash of Eric Raymond's CUPS horror story.)

So I followed standard protocol and proceeded to Fry's to see what was on sale. The cheapest (on sale) laser printer they had also claimed to support "Various Linux OSes" on the box. (Samsung appeared to be the only vendor to even mention linux) Perfect.

A Samsung ML-2010 for $89. And the box wasn't even previously opened (a common occurrence at Fry's). (Though red lights went somewhat off when I noticed one of the distro's supported by Samsung is the now defunct Caldera (don't want to call them SCO cause Caldera was a nice distro in its prime)).

Once at home, I opened the box and tried to use the included software on the cd. It seemed to run ok (as root), but appeared to configure the USB only printer as a parallel port printer. Printing test pages did nothing. Scouring google and linuxprinting.org I found another recent reference for using the printer on Suse, but it didn't appear to help my situation. (I'm running gentoo).

I tried running the web based CUPS, and kde CUPS config tools but none of them listed the 2010 model. I was afraid that my "Linux supported" hardware was not so. (Updates from the Samsung website, flat out refused to run, glibc issue). So, I slept on it.

Later, after reading about other samsung xx10 printers on linuxprinting.org I caught wind of this sentence: "As nearly all of Samsung's low-cost lasers use the same proprietary printer language (during the time it eas called SmartGDI, PrinThru, and SPL) on can also use a free GhostScript driver which Samsung published some time ago for the ML-4500."

I created a new cups printer instance using the ML-4500 (gdi) driver (from the KDE CUPS interface) and lo and behold, it worked! Beautiful prints, and speedy too. (I'm still not sure how others have used the Samsung provided ppd file for the 2010, it appears that it has sytax errors (using cupstestppd)...) Hopefully this winds up being useful to someone.

How to create a passport picture with a digital camera and gimp

2005-08-29T00:00:00-06:00

A quick howto, since I recently had to do this. To create a digital photo for use with a passport you'll need the following:

digital camera
computer
software (I used Gimp though photoshop may work too)
tripod/or person to take picture
white background

Heres a link the department of state that describes how to setup your place to take a picture. Here's another link from Singapore that describes what a "good" picture is (I'm assumming American standards are similar). (It appears that "good" means not smiling or having some "SPECIMEN" text in front of your face, take your pick) Probably the most important thing is the white background, so find some person who's renting an apartment and they are sure to have nice ugly white walls to take a picture against (mine worked perfectly). (I didn't use side flashes as my camera seemed to take a good picture without shadows with the builtin flash).

Once you have everything set up to take a picture, make a really monotonous face and snap it. When you've got one that meets the standards (or has the text "SPECIMEN") you are ready to proceed to your nearest computer.

Load up the passport4x6.png (below, right click->save as..., or drag the link over the gimp tool selector) in your photo manipulation software. The important thing is to keep the size as is (1200x1600), because the red box in the middle will be 2 inches (which is the size you need your pictures to be) if you develop this at 4x6 (no cropping).

Go to the layers dialog and duplicate this layer. Now drag your mugshot into the picture. (It will probably be the top layer.) Move your photo so it is between the red box images (the middle layer) using layer manipulation. (Otherwise when you scale your photo you won't be able to see the red box). Activate the layer with your photo in it (should be in the middle layer). Activate the scale tool (red box in below). Double click it to bring up the scale settings. Make sure that the "Keep aspect" radio button (green circle) is selected. Now scale your face you that it is the same size as the face in the red box.

You are now basically done, you can center the picture if you want. Remove the top layer (red box) and save your image as a jpg. Take to a photo kiosk and print two at 4x6. Cut out 2 inch square around your face, and you are done!

I have an issue with foocamp...

2005-08-27T00:00:00-06:00

I have an issue with Foocamp, and it isn't because I (unlike some of my co-workers) wasn't invited. For that matter I have the same issue with Barcamp (which I may have attended if I wasn't out of state on vacation floating down the Snake River in Wyoming during that time...).

My issue is more of a sematic one. When I think of "camping" I think of carrying around 40 lbs of gear on your back and walking around in a remote pristine environment.

On that note I'm proposing CampCamp. Invitees are required to carry everything they use. The only electronic equipment will be gps and digital camera (or a hacked CVS camcorder). Food will be hand prepared by yourself. The scenery will be inspiring. The days will consist of walking and conversing. Talk about what you want, get a break from your virtual umbilical cords (be they cell phones, blackberries, wifi, etc).

Hopefully this setting will be conducive to planning the next big thing, and you'll come back refreshed and ready to go. Oh there will definitely be a size limit (determine by the size of the wilderness permit obtained), probably between 10-20. After that there will be a series of other camps, CanyonCamp (similar to CampCamp but done in canyoneering style), RiverCamp (done on inflatable vessels), BikeCamp (pedal powered), *Camp...

So will Foocamp please change their name to FooSleepOver...

ps. Yes this post is slightly tongue in cheek, but possible. I know many places that would be awe inspiring, and probably cheaper than a conference (my wife and I hiked to Machu Pichu, and Torres del Paine, and both trips cost less than a conference pass/lodging/travel to OSCON).

You'll never get out of this maze (a new twist on phish)

2005-08-23T00:00:00-06:00

A few years ago Phish released the album Rift. One of the songs was Maze which had a chorus that went something like this:

You'll never get out of this maze, You'll never get out of this maze, You'll never get out of this maze, You'll never get out of this maze....

Well now phish has changed from the name of a jam band to a verb. But this creation of verbs is not necessarily good (like say google). It does not be being bombarded by tortillas during a 2001 jam. Phishing attacks are very common (and according to the security session at OSCON there was no solution for them), so education is important here.

Most geeks know this but perhaps their acquantainces (wives, families, friends don't). So in order to appear like a public service announcement blog, remember to tell/remind all your friends to delete phishing emails. Yes they need to have friendly reminders from nerds who they trust.

ps- yes my wife recently recieved two emails from "paypal", and she was so bothered by it that she deleted her account. (I guess paypal itself (the non-bank bank) is another can of worms).

August Bay Piggies Meeting

2005-08-12T00:00:00-06:00

We had a small meeting tonight. Hasan gave a short presentation on his Really Really Simple Rss browser (45 minute 80 line hack). Details found here.

Previous speakers were awarded their choice of various python books, whick led into a bit of discussion about python books. It seemed that people enjoyed that. Most of the folk seemed to want in-depth books focusing on subjects such as toolkit programming.

The discussion then wandered onto refactoring, what is it, and when do you use it. Most who had refactored in python apparently do it by hand. This may be that it is pretty simple or that the tools are not well known.

This was followed by a brief discussion of OSCON happenings related to python(mainly RoR). Web development talk, django, mvc, templating, ajax, too much choice (vs the "golden path") were are briefly touched on. Following that there was a bit of mingling until we were gently removed from the building. No deep discussion about meta-programming or worrying about if you actually understood the presentation.

SpikeSource Open Testing Contest

2005-08-11T00:00:00-06:00

Calling all testers (and developers). In an effort to boost testing among open source components, my employer is offering $20,000 in awards (as well as O'Reilly books) to Open Source projects who compete in the SpikeSource Open Testing Contest.

Good luck squashing those bugs!

Are there areas where Open Source won't work?

2005-08-08T00:00:00-06:00

One of the biggest shockers for me at OSCON happened during the first day. I was sitting in the second row of the Rails session and I looked behind me to see what type of laptops people were using. I would guess that 80% were PowerBooks. Of the x86 laptops that I saw, I would guess that 10% of those were not running Windows. (So the distribution looked something like this 80% Mac, 18% Win, 2% Linux (BSD?)). (Luckily Miguel was running Linux ;)).

So what is the cause of this apparant mass "switch" to powerbooks? (I bet that most of the laptops were not work laptops, i.e. they were bought by the individuals, so work is probably not pushing mac's (at least not right now while they are still ppc...)).

Are they just sick of Windows, but Linux doesn't cut it? Did the OSCON attendees run linux before switching? Is the competition between gnome and kde marginalizing both desktops? Lack of integration of Linux apps?

I asked Rob Kaye this same question on our flight home, since he has a fancy powerbook. He admitted that he "switched" because of the coolness factor (and utility) of the rendevous enabled apps (like subethaedit). He says he loves being able to "listen" to other sessions of a conference just by watching the notes that people are collaboratly taking (using subethaedit).

I'm posting this because for some wierd reason I thought that most of the people at an "Open Source Conferece" would be running Linux (yeah, I know Darwin is open source....).

Will we see cool/innovative hacks like Greasemonkey and Rails on the Linux desktop? Or will development slow/stagnate as people switch to using Macs as a desktop? Is the desktop just too boring/hard to work on that only commercial devs can do it well? Open Source isn't applicable here? (I'm probably biased since my personal computer has been running Linux on the desktop for almost 7 years now. But maybe if I had a laptop and power management didn't work, I might get frustrated and jump ship....)

IBM opening the secret sauce for Web Fountain?

2005-08-08T00:00:00-06:00

Up above Silicon Valley, IBM has a big research facility. One of the big projects they are working on over there at Almaden is called Web Fountain. Think of it as Google on steroids. A search engine that can determine sentiment (can determine what ambiguous terms on pages actually mean and if the page refers to these terms in positive or negative light (meaning that if some blogger posted "Rails sucks!" Web Fountain could tell that it might actually mean that they are talking about the Ruby on Rails project and the blogger liked it, not that handrails make good vacuums)).

So the news that IBM is open sourcing some search technology that uses "concepts and facts" instead of keywords might lead to some really cool applications. They call it UIMA (it might be different than the Web Fountain stuff (but it says that people from Almaden helped somewhere along UIMA's way)).

Marketing folk would love any tool that can measure "buzz", which is something Web Fountain claims to do. You could see who was talking favorably about your company, and who isn't. (And you could repeat the drill for your competition). Governments would love it because ... let's just say it might help homeland security.

But where's a practical application? Why isn't IBM taking over Google with its new toy? Maybe some inventive folk can hack together something really innovative with this new toolkit. Here's my idea, a blog aggregator that cuts out the cruft.... (when tags start getting gamed it will come in useful)

Trying out Django

2005-08-08T00:00:00-06:00

Having Ruby on Rails pounded into me for 5 straight days (it seemed to be a theme of OSCON) caused me to try out Django tonight.

I've only finished the 2nd tutorial but it's pretty impressive what's in there. The only thing that seems a little wierd is that you can describe the admin view in the model class. I know, even in the FAQ they claim not to be "MVC" but this just seems wierd. I thought modifying the display of items would belong in the controller. (Maybe that's just the evil struts side of me showing..., but I'd really like the model to just be the state of the items, along with methods to alter that state.) If I wanted to reuse the model code to create say a wx app, it would seem kindof weird.

The model code below is what I'm referring to (note the admin section which determines how the admin view is displayed...):

from django.core import meta
#Create your models here.

class Poll(meta.Model):
    fields = ( meta.CharField('question',
        maxlength=200),
        meta.DateTimeField('pub_date', 'date published'), )
    admin = meta.Admin(
        list_display = ('question', 'pub_date', 'was_published_today'),
        list_filter = ['pub_date'],
        search_fields = ['question'],
        date_hierarchy = 'pub_date' ,)

Maybe it will grow into me... On a semi-related note, it is pretty interesting that both Django and Ruby on Rails evolved seperately (django actaully started before Rails (at least the Ruby version of Rails)) and they are pretty similar (you call a command to generate files, structure of apps, DRY).

Gems from OSCON

2005-08-06T00:00:00-06:00

I'm going through my OSCON notes and compiling my notes and a list of interesting websites. The websites will be found here (using my oscon2005 delicious tag):

Yaml

This is the markup language that the Ruby on Rails is using (rather than xml). There are bindings for various languages. The next time I'm thinking about using xml I'll try this.

Dojo

Ajax toolkit. Includes interesting features such as graceful browser degradation. Alex Russell (the main dev) gave a presentation on this.

CodeZoo

Not just java, now has python components. (And ruby too).

Greasemonkey

Attended Aaron's session on Greasemonkey. I've heard of this a bit, but never bothered to check it out previously. There are some pretty cool hacks here. Now the user is really in control of their own web experience (also allows "long tail" feature development).

Diveintogreasemonkey

the book.

Housingmaps.com

This has also been out for a while, but it's a good example of "Riping, mixing, and burning" the web.

Findory

Something mentioned in Tim's(?) keynote, could be a more intelligent way to aggregate blogs since is uses something similar to bayesian filtering to determine blogs/posts you may be interested in. (Organizing the tag soup mess???)

Babynamevoyager

This babynamevoyager itself wasn't presented, but Tim used a tool very similar to this to show the growth in python and ruby and the decline in perl (also a jab at larry wall to get perl 6.0 out). Interestingly enough the author of this tool was sitting behind me in the plane home, and says that they are thinking about open sourcing the tool....

Jonathon Schwartz

no link to him, just an interesting note that immediately after he said how dumb open sourcing java would be (same old scared of forks complaint), he said that "Sun will be releasing ALL it's software as open source ..... or free" (he seemed to remember that he just said he wasn't open sourcing java, and then added the "or free" part). I found it humorous.

Roml

Semasiology of Open Source Part II. Probably the best talk of the conference (which seemed to include most of the "who's who of open source" in the audience). Very funny, again there aren't any links to the slides, but here is Robert Kaye's report.

Roml compared how people of the dark ages learned to read (and that reading was linked to listening, because all reading was oral) to learning to read open source code. He made a funny jab at GPL 3 (because one of the definitions of copyright is to publicly perform, and reading out loud is a public perfomance(or was way back when), exposing web services (on top of gpl'd software) will be participating/running the code, and grounds for releasing source).

(Note: I talked with several members of FSF who also confirmed this (they also said that another feature of V3 will be compatibility with the Apache licences (after they convince Apache to change their licence accordingly). The preceeding sentence is serious.)

Roml also spoke of the importance of the invention of whitespace, which delighted Guido (who was right in front of me), but also Larry Wall found it quite amusing as well (he was directly behind me). He opined that the only difference between poetry and prose was whitespace...

Lang Origami

One of the most fascinating keynotes was Robert Lang, who created TreeMaker to allow him to create complex origami. He showed a few of the very intricate models he had designed from a single piece of paper (such as a rattlesnake with 8000 scales), as well as practical applications (collapsing a 20m lense to fit in a space ship).

Nessus, Absinthe, Metasploit, Wikto, Ettercap, whax

Security/hacking tools discussed by Nitesh Dhanjani (who performs security audits form E&Y). Hopefully his slides get on the website (he said they would be) since they contained some interesting info ;)

IronPython

This was an interesting talk given by Jim Hugunin. Egged on by Miguel de Icaza, he showed some cool integration between Avalon (vista?) where he displayed an xml gui using python. He then scaled it, then rotated it both 90 and 45 degrees. Then he rotated the individual buttons 45 degrees. Then he bound a text to speech event to each button to create a speak n spell.

There was also some really nice debugging support in visual studio, which Miguel claimed should be supported in sharp develop. I talked a little with Jim afterward and I think the main point that he wanted to get across was that you should build ontop of a common vm (clr or java vm) rather than the c python runtime, since you get the benefit of 100s of man years of work. He claimed that without any performance work, IronPython outperformed (c)python in the general use cases. (Guido was hacking away at his new clock during this).

Howtoons

Another cool keynote. A few geeks who create educational comics illustrating how to build things such as a marshmellow gun.

Trip back from OSCON (Rob Kaye's crazy hair)

2005-08-05T00:00:00-06:00

I'm back from OSCON and had a great time. It was fun to meet and talk with so many interesting people. One of the most interesting I met after the conference en route home.

While in the airport standing in the queue to get on the plane Rob Kaye was next to us. He definitely stands out in a crowd, since he has some of the craziest hair I've ever seen. (Here's an older picture:)

We ended up sitting next to each other on the flight home and had an interesting chat about MusicBrainz (his personal project), software patents, the merits of python over perl, the bad designs of id3 tags, and how to motivate people to participate in your open source project. He commented about how to him the only relevant conferences were Etech, OSCON and O'reilly's Music Conference because of the quality of the material and the people present. I'm in the process of compiling my thoughts on OSCON (though I was discussing and learning about geek stuff all week, I was actually using a computer very little (no laptop)).

2005 OSCON Ruby on Rails Presentation

2005-08-02T00:00:00-06:00

I'm up at OSCON and have attended 4 sessions in the past two days. My favorite so far was the Rails talk.

David Hainemeier Hansson (the creator of Rails) did a great job presenting, and I was very impressed by the framework he has implemented. The main gist of Rails, that seems to be apparant throughout the whole app is offering the user an easy and defualt path. Here's one of his quotes (everything in quotes is direct):

"You don't have to follow [my conventions], but it's just so easy to do, why would you not?... [worrying about silly conventions] just doesn't matter that much."

He was very frank in expressing his disgust towards J2EE and other frameworks that are very explicit, noting that rails doens't have "a lot of configuration annoying you". The only configuation on this "convention driven system" shown during the demo was a yml file (no ugly XML here) for handling databases connections. By following David's "wonderful conventions" or "The golden path", there is almost no configuration for rails, everything "just works". There is no deployment step. You have instant gratification and feedback as you change you code. This allowed David to fly through his presentation, creating an AJAX enabled, password protected, blog, complete with commenting in under 3 hours. The ActiveRecord model is interesting, David admitted to me that he thought ORM systems break after you do anything non-trivial. Thus even though there is a "model" class, the actual model's state is stored in the database (and not necessarily reflected by the class).

David suggested that Rails probably worked better for green field development and that "I have no intention of bending backword for these schemes from the 80's" (referring to legacy systems and probably J2EE as well). I think it would probably be pretty easy to hook up legacy systems using web services though. To speak of web services David topped off his presentation with an RSS feed for his blogging tool.

And to top it all off, (and to follow up my previous post), Rails comes complete with an unit testing framework, and browser simulator. Very cool!

I was able to personally chat with David for a bit during a break. He admitted that he was hoping that someone would question him about scalability and how easy and simple it is (sadly there was very little time for questions afterward). He told me over 300 people have committed patches for rails and some have even done it the same day they downloaded rails (without any previous ruby experience), though there are only 4 guys who "know all the code". He was very complimentary on ruby's meta programming abilities, stating that he has an implementation of rails in php that was quite slow due to slow code generation (rather than changing classes on the fly he had to actually generate php files and reload them).

And finally I asked David about Python. He was complementary to the Django project and said if you were doing a CMS type system, it would probably be an excellent choice. Though he admitted that Rails was a better "generic" web development platform.

A common muggle against he who shall be (or is) named Bill Gates

2005-07-25T00:00:00-06:00

Argghh, Windows keeps coming back to bite me. (Or rather me being a "nerd" means I give free support to my friends, who sadly all run some variant of windows)

While having dinner with some friends tonight (who happen to be another uninformed computer user that I blogged about while back), I was asked if I could "look" at their computer.

Basic problem was they had recently signed up for high speed internet, and his wife's computer was no longer able to access the internet. I spent 20 minutes trying to figure out XP's network configuration (I've never owned an XP computer), and I swear that 98 (which was confusing itself) was much simpler.

I was taken through about 20 pages of a wizard before I figured that I was down the wrong path. I was given too many options, with no clear indication of which was the right one (I was trying to either connect to the cable modem directly or use their router). Before I was able to do anything, it was time for our kids to go to bed, so we had to leave. Luckily their laptop still connected...

The point of this post is not to say that I'm a great admin or windows user, but that windows networking is confusing. I'll take gentoo any day. At least I'm given excellent documentation, there is a single text file containing the configuration data, and there is no reboot required.

Easier for the normal (read non-nerd) common muggle? I don't know, I think both would be confusing, but recent versions of Fedora have pretty good gui wrappers around network configuration. At least they don't overload you with wizards. (to prove my blinders aren't too dark, I'll say that wireless configuration (and wireless network detection) has been A LOT easier on windows. In fact I've considered buying a access point just so I don't have to re-install my non-kernel wireless driver (netgear ma-101b) everytime I update the kernel....)

Testing - an afterthought for opensource?

2005-07-20T00:00:00-06:00

From my experience commercial testing of software consists of manual qa (grunt labor). Little is automated. I personally feel that even with a slight investment by developers to create unit tests, that effort would pay off later (especially in the crunch time before a big release where fixing a bug may cause others).

I guess it is hard to justify ROI on testing, thus there is little emphasis on it. Now let's leave the cathedral and head to the bazaar.... Which leads me to another chart.

I recently patched and help port two open source projects, and when I asked the main developers if they had unit tests, they indicated they only did manual testing. This is pretty widespread among smaller projects and I would venture to say many desktop apps.

Yet as projects grow bigger they tend to have some sort of unit or regression tests, and testing improves.... or does it?

Within the past two days I read two articles that highlight the continuum of different levels of testing done to open source software. One was a slashdot article asking if there were any opensource tools available for load testing and qa? It wasn't even given space on the frontpage of slashdot (shows how important testing is to the editors of slashdot...), and recieved very few responses. It seems like most people buy an expensive solution or roll their own. (Much $ spent or much re-creation of the wheel).

The second discussed Coverity's static analysis of FreeBSD declaring that they found a potential of 306 bugs. We've all heard the argument that "many eyes make bugs shallow". There is some truth to that, and you see it in the bigger more widespread projects. As people use the project in their own way, they find bugs, and (hopefully) either report them or send patches. But usually this is an uncoordinated effort.

There is little aggregation of tests (other than unit tests). Also, eyes aren't as able to catch things that tend to get exposed during a good load test. (I'll call the green line the sum of all the testing that goes into a project). When performance, stress, or load tests are done, are they being returned to the project and aggregated there? How much of this is repeated by end users who perform these types of tests? How much is implemented in house (not by the project)? How much time and energy could be saved (or put into new testing) if this information was aggregated and shared (see red line in chart)?

Maybe testing just doesn't itch very much, or we don't like to scratch it.... Any examples of projects that are particularly good at testing?

Alex Martelli's Python Black Magic (July BayPiggies presentation)

2005-07-19T00:00:00-06:00

Last week Alex presented to a pretty big crowd for a baypiggies meeting at google. His talk was originally entitled "Black Magic", but to avoid the bad connotation, he renamed it "Descriptors, Decorators and Metaclasses". A pdf version of his slides can be found here. If you are interested in any of these topics, check out his slides.

One issue that he really didn't talk about was when you would want/need to use metaclasses or decorators. I only scribbled down a few notes. The funniest part of the talk for me was when an obvious python newbie questioned Alex on why classes expose all of their members (private vs public). This clicked something in Alex and he went off. He said something to the effect of "there's nothing stopping you from killing yourself. Your body has no mechansim to prevent that, but it's a bad idea. It would be wasted machinery." He went on to say that one may subvert C++'s private encapsulation through casting and that you should run any untrusted code in a bsd jail or something similar.

ICFP followup

2005-07-19T00:00:00-06:00

I guess I should followup to the ICFP programming competition which held the second and final phase a little over a week ago. The gist of the competition this year was that you implement a project in 3 days, then two weeks later the specs change and you are given a day to change it.

We implemented the problem in python and spent probably 16 hours for the first part and probably 10 for the second. I don't regret choosing python and it certainly wasn't a bottleneck for us.

We spent a good chunk of the final day trying to come up with some inkling of intelligence for a program (the project was to design cops and robbers to defend/rob banks).

To try and visualize some good algorithms we spend probably three hours messing with graphviz. I use graphviz at work a bit and it usually whips out a decent map. But the map we were sending it was a little too big, because instead of spitting out something, graphviz (or dot to be more correct) sat there for about 7 minutes before giving us a map that well to be nice, looked more like modern art. To that end we tried to print it on 20 pieces of paper and tape them together. We ended up with nice wallpaper...

We ended up with a cop that walks around (until he senses a robber nearby then he tries to follow him), and a robber who does a pretty good job of avoiding being sensed. Maybe next year I'll try and recruit an old co-worker (a machine learning guru).

Preaching to the choir

2005-07-04T00:00:00-06:00

I guess that's one of the benefits of the "long tail" of blogging. I can get on my soapbox and speak directly to my audience. I know what to say and how to tweak their buttons. Most of my family and friends stay pretty far away from my blog since it's too "technical" for them. Yet how do I present a version of the message available on my blog (if you grok geek) distilled into more palatable laymans terms? What was the first thing that appealed to me about open source?

When I first heard about open source I thought the idea of give and take was quite compelling. As a poor student, I liked that I could have (for free) the software I needed, the ability to work from my dorm instead of the crowded labs, and that I could tweak my computer. Those are some benefits of open source to a computer science student. But what about a non geek? Say my friend who is a doctor, or my mom who teaches school?

My wife and I were recently able to join my friend and his family for dinner. We don't see each other too much anymore since we live in different states, and he is pretty busy spending 90 hours in a hospital each week doing his residency (not to mention his wife who is clerking for a judge or their newborn child). They asked me about my job and I stated that my company provides enterprise support for open source.

Now I'm going to go out on a limb. The biggest problem facing open source adoption among common folk is ignorance (forget companies with monopolistic practices spreading FUD here and there). Yeah, I guess that's called marketing (but don't ask me, I'm an engineer) or lack of it.

I asked my friend and his wife what they knew about open source, which was very little. But when I told my altruistic doctor friend about the empowering nature of open source, and how it could enable the common user of the computer to use and develop applications that previously required very expensive middleware, he thought that was pretty cool. (I knew that since he had spent a bit of time in 3rd world countries volunteering in medical facilities he would be able to draw some comparisons).

The idea of software being developed for the common good of society resonated strongly with them. Yet they had never really heard of it before...

My mother, who teaches CTE in middle school recently related to me that her whole school district was moving to "open source" software for their office suite. Supposedly she was the only member of the faculty who knew of open source (my mom is known to exagerrate a little bit, but I don't think she's off by much). Luckily I'm her son, so she knew a little about open source and she knew that linux was something to do with a computer.

Does my mom care whether the software she is running is open source? No. But when you tell her that the school district no longer needs to send a big check to a certain supplier of productivity software every year, she will grok that. And it resonates well with her. She would much rather have ingredients for her students to cook with. (And quite frankly she can't tell the difference between two different word processors).

So how do other people present open source to their non-tech saavy friends? How do you change the message so that the choir understands it? I'm not claiming that everyone needs to run linux. It is satisfying though when someone starts to use firefox or openoffice after a bit of friendly encouragement....

Guido's "Building an Open Source Project and Community [around Python]" talk now available in audio format

2005-06-28T00:00:00-06:00

The talk that my first blog post was based on is now available for audio download here. It's a good introduction to what caused the creation of the language as well as a few jabs at other languages.

Enjoy

ICFP Contest Implementation (in Python) Notes

2005-06-27T00:00:00-06:00

So after spending around 16 hours (and writing about 700 lines of original code) on Saturday preparing an entry for The Eighth Annual ICFP Programming Contest, I thought I'd share my notes and some thoughts. A chunk of this is Python specific. (see this thread for metrics of entries using other languages.

It appears that our implementation is smaller than most listed in the thread (and ours even includes test code in the 700 lines, though our implemention isn't necessarily that smart). And it took less time than many others. Too bad we didn't have more time to put smarts in...) The problem consisted of writing robots that navigate a map to achieve certain goals. There are two types of robots to implement, a cop (whose goal is to catch a robber) and a robber (whose goal is to evade cops and steal money). That should be enough context to understand the rest of this post.

Parsing in Python

These robots are controlled by using a simple protocol described by a grammar. In order to parse the text that the games server sends back, we decided it might be a little more robust to use a lexer and parser (I haven't done this since my compiler class in school...), rather than dealing with a bunch or regexes or if/else looping over lines. After a pretty cursory search we decided to use ply. After checking out the example, it seemed that there was a very clear mapping between the grammar described in the problem and the PLY implementation.... But using ply was probably our biggest mistake (and time sink) of the day.

Don't get me wrong, maybe we are just stupid since the home page states "PLY is extremely easy to use and provides very extensive error checking", but we had a few issues that got us hung up for a while. Since we only had the day to program we didn't have the luxury of emailing the mailing list, and there didn't appear to be a FAQ anywhere.

I'm sure if we had had more time, we could've figured out our issues. Some more gripes about PLY is that is isn't "pythonic" (a quick attempt to use the lexer and parser in a more object oriented manner failed) and Mr. Beazley is doing some magic behind the scenes that made implementation debugging a little painful.

After realizing that we were wasting cycles my partner and I decided to each go and try and implement a new parsing solution (we had basically been pair programming til then). I ended up trying pyparsing. The initial drawback to pyparsing seemed to be that instead of having a straightforward copying of our grammar, we had to convert all of the grammar tokens into objects.

Also, my quick browsing of the examples provided seemed to not provide me with enough understand of pyparsing and its objects to implement our grammar. The examples found in the article by Dave Kuhlman, gave me the examples I needed.

In a few minutes we had an implementation to parse the most complicated message. A possible drawback to pyparsing is that it appeared to be rather generous in what it accepted (which didn't matter in our case since we were guaranteed that the server wouldn't send back invalid content).

Flush stdout

We were also having other communications issues including a broken development kit for windows (it worked on linux though). Our robots communicated with the server via stdout, and stdin. A small bug that bit us was forgetting to flush stdout to talk with the server. Rather than reading our message the server would time out because we didn't flush the buffer.

Old state machines

We first implemented a simple robber robot because he had very little state. The cop had a bit more state, but the transitions from state to state were pretty simple and there were few branches. We could implement a big switch statement (bunch of if/else in python) or try to use a state machine. After a quick google, we choose one out of the two we looked at. The two in question were one from the ActiveState Cookbook and one from a 10 year old email (written by Skip Montanaro). We choose the 10 year old one because the cookbook implementation had a comment to code factor of about 10 (which made it look really complicated). Skip's implementation was really simple and we probably only changed 3 lines of it and it worked.

Executable Pseudo Code

To implement some of the graph algorithms I brought my Cormen Algorithm behemoth along. The implementations proved very simple. I would dare say that there was a 1 to 1 mapping for pseudo code in the book to python code (1 line in the book translated to 1 line in python).

unit tests

Along the way we created unit tests using unittest. These proved to be valuable as we changed occasionally the interfaces to some of our classes, we could run the tests and see what broke.

Pair programming

For probably 13 of the 16 hours we were pair programming. I think that helped us is a few ways:

1) We were both able to understand all of the code

2) We could catch errors along the way before executing as the 2nd pair of eyes can become a virtual compiler.

3) Using python we could brainstorm and implement really quickly to try out our ideas.

4) Productivity and motivation. Having someone right next to you tends to make you pretty productive and in this case I feel that 1+1 > 2.

We still haven't implemented much of the brains, and that may prove to be pythons downfall in this whole exercise since the robot has a time limit to submit their move. But for implementing a basic robot, python and the existing code libraries for it proved invaluable.

Linus on Open Source Evolution and Competition ... and Robbins joins MS

2005-06-21T00:00:00-06:00

I recently read an interesting interview with Linus. While I would recommend reading the whole interview, here are some highlights that tend to support some ideas I've been throwing around on my blog.

Open source programs have clearly made some great advances in the past few years. We've seen it with Linux and Mozilla. What sort of future do you foresee for open source? What do you foresee happening to software vendors tied to proprietary software models?

The power of open source really lies in various groups improving and building on an increasingly bigger existing base and slowly turning that base into commodity. One of the keywords here is "slowly" - it's by its very nature pretty evolutionary, i.e. it takes time. It's, in my opinion, also pretty unstoppable, but the process definitely makes it possible for proprietary vendors to generally take advantage of the open source commodity base, and continue to be proprietary "on top" of that base. ...

What will happen to the PC world if Microsofts market share should decline? What will the technology landscape look like?

If Microsoft loses its dominance, that's likely a good thing for the market in general. Again, see the IBM connection from a few decades back: More open competition tends to make the market not only more lively, but also tends to grow it. So the notion that many people seem to have that the PC market lives and dies by MS dominance would seem to have no basis in reality. If anything, near-monopolies tend to stifle things.

Linus describes the progress of open source software as improving "slowly" and that is generally the pace, though I have highlighted a few catalysts in past posts.

I have also had mixed feelings over that past week with news that Daniel Robbins (the creator of the Gentoo distribution) has gone to work with Microsoft. (I'm writing this from a gentoo box...). I guess one possible fruit of this move might be that Microsoft (with the help of Robbins) provides some sort of new open source catalyst (and/or competition) to further improve the collective software pool...

Somehow I don't think I'll be performing a stage one longhorn install anytime soon.

BayPiggies meeting with Drew Perttula

2005-06-09T00:00:00-06:00

I attended the BayPiggies meeting where Drew Perttula talked about his lighting system, light9 tonight.

The crowd was a little smaller since there was no free dinner, but there was a pretty good turnout. What a small world, Drew works two buildings away from SpikeSource at PDI, and I play pickup ultimate with his coworkers and cousin during lunch.

Drew is a very good speaker and did a great job with his presentation as he was willing to discuss areas of python that weren't only limited to controlling lights. Light9 is a series of python programs he uses to control theatre lighting in realtime. He has written a series of programs that communicate over XML-RPC (twenty times per second) to update the lighting.

I'll highlight some of the gems I gleaned. He raved about the robust nature of Twisted noting that "it doesn't get in a bad state". He is using twisted to talk XML-RPC (see my previous Peter Yared of ActiveGrid post about the merits of XML-RPC) to communicate between his programs. One of the more interesting facets of his presentation was his series of UIs to control the lights, and the innovative yet distinct manner of each of them. Another was a keyboard to light mapper and the user could hold down keys that were bound to different lights to change their brightness. Another was a map that depicted the stage and where the lights should hit it. The user could select spots on the stage with the mouse and then drag-click to brighten or dim the lights. Another was a waveform analyzer (think audacity for lighting) that let Drew change the lights based on the music he was playing (as well as combine it with other filters).

The most basic control was just a list of lights and you could select them and raise them on the fly. All of these programs could be running at once (all talking xml-rpc) to the server which controlled the lights. Pretty cool!

Pydispatcher allows him to set signals and slots. He was using this as a mechanism for the model notifying the view (in his case tk) of changes. I'll have to check out pydispatcher as I haven't heard of it before.

There was a brief discussion of darcs and its merits over cvs and subversion. One of Drew's "slides" contained wonderful ascii art that he generated in everyones favorite operating system emacs (using "artist-mode", something else I've never seen before).

Drew also discussed his deep desires to rid the world of load/save functionality and his desire to automate the world beginning with his home. Thanks for the good insights Drew, see you around the campus.

Interview with KDE developers Ivor Hewitt and Allan Sandfeld Jensen (mergers of Apple's ACID2 patches)

2005-06-08T00:00:00-06:00

Previously I've blogged about Safari's fork of KHTML and the expertise of Apple's browser developers. With the recent news of Konqueror becoming ACID2 compliant, I yearned for more info.

I just conducted an interview with Ivor Hewitt and Allan (carewolf), the developers who merged in Apple's recent ACID2 changes into Konqueror. Read on to get their "inside" feel for the situation between KDE and Apple.

Thanks Ivor and Allan for your excellent work and for responding!

Please describe yourself. What's your day job?

I: I'm from Surrey, England. Currently my day job is as a freelance developer working for a London investment bank on their trading systems. Unfortunately that, and a daily commute, does mean that I have a lot less time available to work on Open Source projects than I would like.

A:I am computer science student at Copenhagen University, I am starting to write my master thesis this summer. I have mostly supplemented my scholarship by teaching/tutoring in OS/Kernel programming and Compiler Theory.

So, do you mostly work with C++?

I:Being freelance I work on pretty much anything - Java, C#, C++, Perl, but C++ is certainly my preference.

A:I do if I can. Academically I also write quite a lot in SML, but mostly because I've never taken the time to learn OCAML. Besides that I dabble in Perl, PHP and occasionally C.

How did you get involved with KDE?

I:I've always used KDE as my main desktop environment on Linux. I've tried a few others but I just like the "feel" of KDE. I honestly can't remember the exact version of KDE I used first, or for that matter even the first version of Linux, although I did find a post I made talking about upgrading to kernel 0.99.

A:Well. I had a job in 2001 where part of my job description was monitoring the open source projects we were using (Linux kernel, Netfilter, PostgreSQL). At that point I got involved in the linux kernel. Since I was using KDE at the office I soon converted my personal mail from Windows to Linux and switched to Linux as my main desktop in late 2001.

I started maintaining a user version of KDE at university, and made patches for running KDE on Solaris, Alpha and HP. I gradually found little things in KDE I wanted to improve and at some point I just became a KDE developer.

What were your motivations for working on KDE?

I:Following a few problems on another project I had been involved in I was looking around for a new challenge. Initially I decided that I wanted an Firefox AdBlock style feature in Konqueror simply as an exercise to get familiar with the code, I wasn't sure if I'd be able to do it. So I was rather surprised and pleased when I got it working in a matter of weeks. I was also really pleased with the response I got from the developers and the warmth with which I was received - that's a great motivator and makes you want to get involved.

It was only after I'd got AdBlocK working that I found that it had been outstanding in the bug database for about five years and had something like 2000 votes. It just happened that at about the same time I was doing this Dave Hyatt announced his Acid2 pass, it just seemed like an ideal opportunity to work on another well constrained problem that would allow me to get more familiar with the code.

A:That is simple: It's fun. Also you get real results, and can make big differences, where other projects like the kernel, mostly provides small improvements in very specific cases.

Why Konqueror? Isn't Firefox good enough?

I:Konqueror integrates nicely into KDE and it's much more than just a web browser. You're not going to get me saying one is better than the other :). Firefox is good, I use nothing else on windows, but its KDE integration isn't great and I find Konqueror snappier and easier to work with.

A:In the beginning I actually choose to use Konqueror, because I just couldn't stand the way Mozilla renders Slashdot and many other pages. Back then I had no idea why, but today I know it is only because Mozilla defaults to serif fonts, where Konqueror, like IE default to sans serif fonts.

A funny thing but if Mozilla didn't use those serif fonts by default, I probably wouldn't be a KHTML developer today. Later I've just stayed with Konqueror for its speed and features, but those takes time to discover and appreciate.

How long did it take you to feel comfortable with the code base?

I:Well not that I can say I'm 'comfortable' with the code base or understand how it all works yet, but I found it extremely easy to navigate my way through the code and figure out how it hung together in just a few evenings. That is a credit to the quality of the code and architecture that's been put in place. Given how little exposure I'd had to the KDE code it was very rewarding to be able to make a difference so quickly.

A:There are large parts of the code base I still don't understand (like DOM and ECMA). I would say that for the areas I've used it hasn't taken more than a few bug-fixes or experiments to understand it, and usually you can understand it well enough to make a bug-fix within few hours. When I started developing KHTML i August 2004, I didn't know any HTML or CSS besides basic markers, I learned much of HTML just but reading the KHTML code. It is that neat.

What does your development environment look like? Are there any tools you use to make you more productive?

I:My development environment both real and virtual is a complete mess. In the real world, at the moment I have three disassembled Vaio's and two mini-itx machines in pieces on my desk along with a new toy - a surface mount removal tool and my main PC.

As for my virtual environment, I use loads of different tools depending on what machine I'm on and what I'm doing. Reworking existing code I use emacs, minor fixes I'll use vim, writing new stuff from scratch for fun I'll use KDevelop. If I'm reverse engineering then I use IDAPro on Windows.

Most of my development, however, I do on my trusty but slow Vaio on my daily commute into and out of London. As for the tool that makes me most productive.... that'll be distcc in the evenings. I'll still be working on the Vaio, but compiling remotely on two P4's "it just works"(TM).

A:I do almost everything on my laptop hooked up to 21" monitor. There is nothing like lots of Konsoles and KWrite windows to speed up productivity. I stay away from most fancy editors, I like to be in control of what I am working with. If nedit worked in KDE I would probably still be using that.

To make Konqueror ACID 2 compliant, what did you do? Did you merge Apples changes in, or rewrite? (Or some combination?)

I:As Allen said in his announcement: "Roughly half of the Safari Acid2 patches could be merged in some form or another, and another two bugfixes could be merged from Safari 1.3 once found and isolated. The rest was written from scratch. All in all not a bad reuse of code, although slow."

So a combination of both, most of the time was consumed understanding the reason for some of the Apple changes. On the other hand the fact that David Hyatt had already identified what areas were causing problems no doubt saved time. Of course the fact that you know that success is actually possible helps a lot.

A:I started by looking all the patches over and merging what made sense. None of them could by auto-applied, so I applied them manually changing and expanding various things underway. Two of the bugs turned out to have been already fixed in KDE, and one fixed a bug we had never introduced. One patch was entirely rewritten, but with the same idea. When that was done there was still many holes in smiley, and I left the work to focus on my university project.

After a few weeks Ivor turned up and showed much interest in fixing the rest. I gave him a few hints and let him work. After a few days it turned out some of the issues where more problematic(interesting), and I just couldn't keep myself from trying to solve them (I am sure this will cost me a grade in my project). When I joined, Ivor and I fixed the remaining bugs in less than two days.

How would you compare Apple's code with KDE's code?

I:Can't say I have an opinion on that at all really. Clearly the majority of the code I was looking through was KHTML code but with just the occasional Safari change. Ok I do have an opinion I dislike their C++ indentation style difference. :)

A:Not bad. In the most recent release the Safari 1.3/2.0 code, they have actually started to make several clean-ups. However it also has obvious hacks here and there to fix specific webpages.

Did you have any interaction with Apple?

I:I had a few chats with David Hyatt on IRC, but other KHTML developers have had more contact.

A:Not until after the first two Acid2 blogs. They contacted all recent KHTML developers to ask how we could improve the relationship. It was here established very quickly that Apple would open their CVS, but they wanted to announce it at WWDC. Apple also had an interest in KSVG and since they didn't have the manpower to write a SVG render themselves, they wanted to get our code base close enough that they could adopt new KHTML technologies easily. So now were are aiming for a win-win situation and talk regularly on IRC.

What is your take on the idea that KDE could "refork" Apples work and use that instead of the current KHTML code?

I:Events have overtaken your question here. :) With the announcement of Apples WebKit Open Darwin project it should be much easier to be able to trace and understand changes in Apple's code.. Personally I'd say that throwing away KHTML and starting again from WebCore isn't necessary and just doesn't make sense, a lot of the changes made by Apple have been done out of necessity due to their different underpinnings, aside from that another fork would again be just that, it would very quickly diverge with the different demands and pressures that a company like Apple has to those of KDE.

With Apple's announcement of Webkit, do you know if you or any other developers will be tracking changes to Safari and merging those more quickly? How will this announcement affect KHTML development?

I:It's too early to say what's going to happen. Yesterday it was crazy, at one point there were 120 people in the #webkit IRC channel. It will be interesting to see how things progress over the next few weeks and months as things calm down. At the moment I believe we still don't have access to the historical bugs database, but I think that's being addressed. That will make the most difference, seeing a change in the code that says this addresses issue #xxxx and then being able to actually read the bug report and find out what the fix was for and seeing a test case, that will be useful.

A:I started to track their commits yesterday and have already merged the first patch (implementation of the Mozilla extension DOMParser) a few hours after it was committed to WebKit. When I get more time during the summer holiday, I plan to skim through more of the CVS history and pick isolated patches that will improve Konqueror and close the source gap.

Apple's latest announcement

2005-06-07T00:00:00-06:00

Not quite sure how to react to this. The only reason for the "switch" as I understand from reading the keynote is a speed bump. Since MacOSX will only run on Apple sanctioned x86 hardware, is there really a difference if it is not ppc inside? There "shouldn't" be a difference, but...

There could be some residual effects. It might cannibalize PPC sales for the next year (and I thought apple was a hardware company). Soon after the release, hackers will probably make osX work on most any x86 box. That should scare MS. I don't know if Steve wants to stop being the BMW of OSs and become the Toyota of OSs (I'd probably buy a copy if it was released generally for x86), but having Dells ship with OSX would be pretty cool (I wonder if the margins of OEMing OSX would be greater than the huge markup to the smaller market?).

Linux/Open Source implications.

I purport that competition is good (it's sad it a way to lose competition on the processor front too). More widespread use of a great OS should inspire open source developers. Will there be some sort of "wine" for apple x86 binaries? Will linux-cocoa binding be created/reversed engineered, thus allowing some great applications to be easily ported to linux? The end of Pearpc and Cherryos? ;) Will I be able to virtualize OSX and MS on top of Xen with the new Vanderpool chip?

SDForum discussion with Peter Yared of Activegrid

2005-06-06T00:00:00-06:00

I attended a discussion by Peter Yared, CEO of Active Grid sponsored by SDForum tonight.

It looks like he has a pretty revolutionary vision of app servers (the java folks might be a little disappointed). Among current IT folks there is a groundswell of support for using LAMP (Linux Apache Mysql Python/Perl/Php). He called this "scaling out" (which was the same term Marten Mickos was using at MYSQLUC, instead of "scaling up" (big iron servers, with complex J2EE apis).

He cited many web "2.0" companies as leading this trend, among them Google, Yahoo, Amazon and Sabre (travelocity). These forward thinking companies have "smart" people making this technology work. Active Grid's vision is to bring this to the common programmer. To illustrate this further he gave the example of a visionary who combined google maps with craigslist housing, stating that SOA (service oriented architechtures made this possible), but then lamented that these cool applications are not being brought into the enterprise (I'm assumming his reasoning is that there is complex integration and apis to deal with).

Peter made an interesting point in saying that the lingua franca of machines today is self-describing XML over HTTP (and went so far to say that this "dumb" protocol would be the last protocol that a computer scientist would design if you told them to make two computers talk to each other). But it works. Active Grid is building ontop of this protocol using open standards (XML, XML Schema, XForms, XQuery, Xpath, BPEL and DHTML).

They have an open source editor written in wxPython that talks to dbs (or Rest, SOAP servers) and creates XML Schema for you. You then graphically create bussiness rules using BPEL (and you can drill in and write code in python, php or perl if you want to). In the future they will allow paying customers to create "adaptive applications" where one can change caching and session mechanisms on the fly at runtime based on user profiles. Sounds pretty cool.

That was the basic idea of his talk. A few other notable things that he stated were: He didn't think Open Source companies that were commoditizing current commercial were that interesting, and stated that Activegrid was a technology company. Active Grid has open sourced a few projects, and have even received community bug fixes. Peter attributed this to "making something that is good", which naturally will pull in a community. From someone who has had most of his career in Java, it was interesting to here him say that Linux is closer to the "write once run anywhere" platform than Java is. It's free and you aren't locked into a vendor, you can migrate from Redhat to Suse to Debian.... He also stated that using python and leveraging pre-existing components allowed them to build their technology very quickly.

r2d2, the ugly robot that could

2005-06-05T00:00:00-06:00

I just saw Revenge of the Sith (yeah I'm a little slow). All in all, I thought it was pretty good. Just have to say (and it's been said before) it's a good thing r2d2 is such a good hacker.

That little guy can fix spaceships in flight, attack others, and interface with any onboard computer system (either he is very standards compliant (as are all the other ships in the galaxy) or he knows how to hack). I won't say much more other than it was interesting to note how r2 got a few new features, since he (and 3p0 for that matter) are so "old school" compared to any of the new robots. I just went to imdb view the discussion boards and they now require accounts to view them. How lame, what a great way to kill architecture of participation....

So even if you didn't really care for the movie you learned an important lesson, standards compliance is good.

Google Recruiting through Subsidizing Open Source?

2005-05-31T00:00:00-06:00

It appears that Google has a new way of recruiting, paying students to become open source developers.

I'm thinking about this Summer of Code thing and the only reason I can think of for doing it (other than not being evil and maitaining geek karma) is for recruiting. If google is going to conferences such as MYSQLUC for no other reason than to recruit, why not try a new method? Perhaps they are apparently running out of good resumes. And if they maintain their geek credibility maybe eventually geeks will decide to leave their employment and work at google.

What does Google have to lose? At most $1 million. In return they get great marketing, and 200 potential hires. We get potentially 200 improvements in the open source world. (Go mono! err I mean go harmony!) (Note to self, write about open source innovations through commercial companies...)

SpikeSource TestFest

2005-05-31T00:00:00-06:00

For those interested in testing and Open Source, my employer (SpikeSource) is throwing a TestFest on June 17th at our offices in Redwood City, CA. Our Test Upload Service allows you to test your code (open or closed source) against various platforms. Come eat our food, drink our drinks, try out the testing service, talk with developers or see demos.

Please visit the TestFest page and register if you are interested, or the SpikeSource website for more information about SpikeSource.

Expertise and Open Source

2005-05-24T00:00:00-06:00

[forgive the lack of charts, my blogging service is in the middle of an upgrade] I've been thinking about another driver of features in Open Source software given the recent noise surrounding the harmony project.

I'll get back to harmony in a minute. First let's examine another Open Source java project, lucene and talk about what makes it successful.

Lucene is an open source source engine that is very popular. You might not have heard of it but you have probably used it. Wikipedia uses it, as well as a whole boatload of other sites that are listed on the lucene website, and I'm pretty sure a whole barge full of commercial sites are using it but not telling you (which is perfectly in line with its license). Why is lucene the de facto open source search engine (and possible the de facto commercial behind the scenes search engine)?

Being open source doesn't hurt, but when it is an important component like search, having features is also very important. And who better to implement a search engine than Doug Cutting?

What is it about Doug that makes him such a good open source coder for search engines? Well he has done it before. He did it at Parc, Apple and Excite. (He's also leading development on an open source replacement of Google called Nutch. See this interview for more details about Doug)

Lucene is a very nice piece of software that does what people want (well barring those that have ported it to their favorite language). It doesn't make sense for someone else to re- implement the wheel, when Doug already knows about benchmarking search engines and can design one from the ground up that will be fast because he's done it a few times before. So that coupled with a pretty strong community around lucene leaves it as a prime example of an open source gaining features due to expertise in an area.

Whats another example? Well I guess I can mention Safari, since I haven't for a while :). Sure Safari is also a "fork" but let's ignore that for a while. The main developer of Safari, David Hyatt, worked on the Netscape, Camino and Firefox browsers prior to working on Safari (and KHTML). The man obviously knows his stuff, and hence MacOS users get a browser that is compliant with the latest web standards (plus it's open source as well).

Whoever hired Hyatt was pretty smart. I'd bet than even if Apple didn't get the head start they did from KHTML and built a closed source browser it would still be very good. Makes you wonder if MS would try to pluck him off to work on their latest incarnation of a broswer.

Let's tie this all up. Will Harmony succeed? I think that lies in the hands of IBM (why IBM? well they just acquired a company called GlueCode who is pretty involved in the Harmony initiative). Will they allow their team to work on the JVM (apply their expertise with such fun things as JIT (which you probably don't/can't code in your spare time)) or donate code to it (ala eclipse)? If so, Harmony could be done relatively fast and would be a boon to Linux, non x86 platforms and IBM as well. If not it could be years before harmony gives us a good implementation.

Python Challenge the greatest thing to happen to python since ruby on rails?

2005-05-24T00:00:00-06:00

So I've been spending a little bit of time doing the python challenge. It is interesting and somewhat addicting. A few other people at my work have also started on it, and it seems to have picqued their interest in python.

This seems like it could very good marketing for python. Is it gaining much traction among non-pythoneers? I noticed the slashdot mention on sunday (sadly it wasn't on the front page). Are many people from other language camps trying out python via the challenge? (Or are they just doing the challenge in their own preferred language to prove a point?) Oh yeah, pypy could be pretty cool too, just haven't had any time to play with it ;)

Bay Piggies meeting with Alex Martelli

2005-05-19T00:00:00-06:00

I was lucky to get a seat (and dinner) at the standing room-only presentation by Alex Martelli tonight. The talk, which was a repeat of his PyCon talk, called Python Design Patterns and OOP, was hosted by Google. His theme throughout the talk was that of delegation in OOP (object oriented programming), and who gets delegated what when programming. I'll provide an overview of what I gleaned. Design patterns referenced here should be available in the online Python Cookbook or the dead tree version which was authored in part by Martelli.

Hold vs Wrap

Holding and wrapping both hinvolve encapsulating "sub-objects". They are distinguished that in the former the api intends for the user to call methods on and access the sub-objects directly, while the latter holds the sub-object in a private name and delegates calls to it through methods. Use wrapping as it allows for flexibility in swapping out the implementation as necessary. A good rule of thumb is if you have more than one dot (period) when accessing the encapsulating class, you are probably using "holding" which leaves the user hard- wired into an implementation.

Wrap vs Inheritance

Wrapping has an advantage over inheritance in that you can hide and restrict access to members. (Don't provide the user with too much power if it's not needed). Most oop programmers think that polymorphism is achieved only through inheritance, yet this is not the case for python. Because python uses "duck typing" (if it quacks like a duck, it's a duck, if it floats...), and polymorphism is based on signatures not classes, one is able to wrap and still achieve polymorphism.

Self delegation

Classes in python may delegate implementation within themselves by separating "hook" methods and "organizing" methods. An example Alex gave is the Queue class which has a "put" method that takes care of thread sychronization while accessing the queue (an organization method). Within the put method there is a call to "_put" which is a hook method. One may subclass or override the _put method if they want to control the storage of the item being "putted" but not worry about the threading issues.

The UserDict class also allows for self delegation as the user must implement the "hook" methods, but have the basic skeleton for a class, and don't have to implement that plumbing.

Other Design Patterns

Alex discussed a few more patterns, namely Strategy (useful to use in place of a flag when implementing different levels of logging/debugging) and Null Object (replace the "if x is None:"). To cap it all off, Alex presented the difference between Singleton and Monostate (borg).

He lamented that Guido was not present to be convinced of the Borg's usefulness. (See the link above where Alex has commented about the differences between a singleton and the borg and why you probably want to use borg). All in all it was a well presented talk of pretty sound pythonic programming practices. I'm looking forward to his "black magic" talk in two months.

Screendoor Security

2005-05-16T00:00:00-06:00

I just had a wierd experience. While rocking my son to sleep I saw something jump across the floor. The lights were kind of dim but I was pretty sure it was too big to be a spider and too small to be a rodent. Upon closer inspection it was a amphibeus domesticus, the common house frog;) Actually, it was an intruder and must have entered through the screendoor that has a small two inch slit in the bottom of it. I'm assumming my soft whistling attracted it, but I'm still amazed that it even got in the screen door (the hole doesn't seem that big)!

While on the topic of security, I'm currently reading Hacking: The art of exploitation. Even though I haven't coded in C for a couple years (since school basically), it is eye opening to see how insecure it can be (or how easy it can be to hack a computer). I'm amazed! I'll try to write up a review when I finish it.

In the meantime, limit use of suid. Or use a more "secure" language ;) With respect to our new pet, he's back outside. Tomorrow we'll be asking the apartment manager to replace the screen...

Freedom forking for features in Open Source

2005-05-12T00:00:00-06:00

Considering the recent discussions about KHTML and its fork found in Safari, I thought I'd post about forks.

Are forks good or bad? Yes. One might argue that people should collaborate rather than fork. But doesn't most open source development actually happen by forking? (At least at a micro level) Developers pull their own source tree and work on it (essentially creating a fork) and finally (hopefully) submit changes back to the main line. Forking is a unique feature found in open source licenses and can prevent vendor lock-in.

What are some good forks? Two I can think of are xorg and inkscape. The xorg fork was sparked by a change in the license of xfree86, but the fork had been brewing for a while. Though licensing was an issue, the bigger issue appeared to be that those who had commit rights to the tree refused to commit new features developed by others (even though they were useful). Xfree was not just stabilizing it was becoming stagnant, and it's developers disgruntled. With the xorg fork, the codebase is open for the distro engineers and others the commit fixes and add improvements. Xorg appears to be moving much faster now after the fork. And users have moved on with them.

Inkscape is a similar story. It is a SVG editor that was forked from sodipodi. Work on sodipodi had slowed down but inkscape developers not only had a different vision for the end product, they wanted to implement it in a different language (c++ instead of c). Inkscape quickly added features after the initial fork and has been releasing often and early ever since. The capabilities of inkscape have grown in leaps and bounds. In the meantime, sodipodi hasn't had an update in over a year (they seem to be more focused on flag clipart).

In both the case of xfree and sodipodi, it appears that development has slowed since the fork. Why? Developers and users (and distros) jumped ship to the more featureful (or insert your favorite reason maybe it's better licensing, friendlier developers, better gui, etc) fork. So apparently the fork was bad for the original products. On the other hand these forks were a net positive as they caused an infusion of features and developers to the new forks. (That might have something to do with the leadership and outreach of the new project maintainers).

Forking can cause an increase in features as well as user base, and can be "good" for open source. The KHTML fork doesn't quite fit in here since it currently isn't even compatible or usable on linux. KHTML cannot at the moment drop it in Safari's renderer as a replacement. Possibly in the future, with help of Apple it might become a reality. Or both sides can maintain their own fork (which is useful (though arguable less so) since both code bases are open and developers can gain glimpses of insight from the other's code even if it doesn't patch cleanly). Any other projects that are "good" forks? What causes them to be "good"?

Fear and contribution to open source

2005-05-05T00:00:00-06:00

I was talking with an Apache contributor today. He was lamenting that more of the users of his projects could contribute back, but did not. I asked what he believed were the reasons. We discussed two: One was work. People or their employers did not feel comfortable giving back. The second I thought was more interesting. It was fear.

Users are afraid of rejection that might come when they submit their own code. When they are possibly able to contribute to a project (especially one that has well known developers) they are afraid of the harsh criticism of their coding abilities. He made one more interesting point stating that when those users do contribute, their contribution usually ends up being quite good (often better than long time contributors). This is not because they are better coders per se, but caused by the effect on the project community. As a first time contributor submits his code, the other members of the community are more apt to review it and improve it. Casual lurkers on the mailing list learn from this action and on the whole the community around the project is improved.

Free your mind with Swing (and freemind)

2005-05-05T00:00:00-06:00

I feel sorry for Java Swing apps sometimes (I don't want to get into a discussion about bloat here, cause I'm not really talking about it). Python is immune from it because it doesn't really have a gui library (I've even heard Guido say that guis are provided for better by 3rd parties than being in the core libraries), unless you count tkinter apps (though many distros don't even ship with that enabled).

Why do I feel sorry? Because Swing apps are not native to any one operating system, you don't hear about them very much (java editors/ides might be the sole exception). Even the useful ones. They aren't marketed. No one wants them. Open source "purists" neglect them because java is not "free", or the widgets don't look right, or the integration with the desktop is missing. Non-technical users don't want to install a runtime engine.

Just the other day at work, someone asked me for my thoughts on a subject and I sent a screenshot of a mind map I made with freemind. A few people asked me later about the tool. Freemind is a useful tool for brainstorming or outlining (could use more polish but as is it is functional for me), yet not shipped with Linux distros. It also runs on Macs (which lack mind mapping software) and windows (if you don't want to pay for mind mapping software), and anywhere else you have a jvm 1.4+.

Any other useful swing apps out there that more people should know about? What about you Apple people out there, do you use swing apps? FYI (there are "native" kde mind mapping tools, kdissert and vym if you don't want to pollute your desktop ;) )

Flattery and Open Source

2005-05-01T00:00:00-06:00

So Apple just released their latest OS.

It has all the newest bells and whistles. Which it should since Apple has a lot of really smart people working for them. They certainly innovate a lot over there in Cupertino. And guess what, even us non- apple users benefit from them, because others will imitate the useful features.

There I said it. Microsoft will do it, open source will do it. I'm not saying that these other two groups don't innovate, because they certainly do. But they also try to borrow the best from others as well. In my quest to understand drivers of open source features, here's another chart: I'm not including adoption this time, since I really have no clue about it (and I don't think it really makes sense in this case).

The chart illustrates a specific example of a theory that I'll call "feature flattery". It compares an application of OS X that I find really useful, Expose and an open source attempt at a clone called Kompose. (Why do I find them useful? I guess cause I'm messy [a slob]. Yes, I know about and use virtual desktops, but my machine's desktops are still a mess. Some might claim that Expose is Apple's answer to virtual desktops, but for me both are nice.

My work machine probably averages about 20 xterms and 15 browsers(not counting tabbed pages) and 10 emacs instances at a time.). Hopefully the graph makes sense. The main idea is that said commercial company comes out with a polished implementation (it jumps in features every major release or so). I assume Hans Oischinger tried Expose, found utility in it and began to create a basic open source implementation (which rapidly grows in functionality until it has the authors desired feature set).

Does the clone have all the features of the commercial implementation? In this case no. Is the clone useful? To me, yes. Will the clone get better? I don't know, the new features of X (windowing system not os) might make the implementation prettier (with windows flying around and dynamically resizing). And that's "feature flattery", that some spikes in the features (and even creation) of open source software occur quickly and rapidly after a mature commercial release (or a detailed description of said feature).

Kompose's website is currently down but according to kde-apps.org, there has been a little progress recently. But the current features work for me. Thanks Apple and thanks to Hans Oischinger.

Now when will KDE pull it in?

Forking the LGPL into the BSD

2005-04-28T00:00:00-06:00

Bear with me for a moment. Let's say I have a fruit tree that has pretty good fruit on it. It also never ceases to bear fruit. And I give away the fruit for free out of the goodness of my heart. (I'll even let you take my fruit and dip it in chocolate and sell it). And being a hardy tree, anyone can take a branch and put it in their yard, provided they also let people take the fruit. Sound familiar? Now let's say you transplant a branch of my tree in your garden. Secretly 6 months later, you allow others to take the fruit. Only now the fruit is a little different, maybe a little tart for my stomach. Plus it also has a hard shell on it. In fact even make use of the fruit I have to use a tool that you sell to me (it's a really nice tool and a few people are willing to pay for it, not only does it work nice it looks pretty too).

Sound familiar? Something similar to this happened with KHTML and Safari. The basic story being Apple hired a really smart guy (Dave Hyatt, who worked on mozilla) to create a web browser. The decided to use KHTML instead of gecko for a starting point, but instead of working with KDE, Apple has worked in a forked environment. Just recently Hyatt has posted in his blog how recent changes to Safari have made it the first browser to pass the acid2 test. No mention of konqueror which renders the test as shown below (version 3.4).

Who is at fault? KDE for not having dedicated KHTML developers to track the changes? Apple for moving (forking) too fast? I don't know but there are opinions from kde and apple flying around. This appears to be a case where open source is not really benefitting as intended. I guess it could be worse, Apple could not release any code at all, but it could be a lot better. I'd love to see both Safari and KHTML helping each other out, rather than just KHTML giving Apple a big jump in developing a web browser. Is Apple really that afraid that KDE will take over it's market share? It appears to me that Apple has forked the code of KHTML so fast that KDE developers cannot merge the changes back in, essentially changing the license into more of a bsd style. The scary thought is that some big company might pull a similar tactic on a bigger piece of more important code (kernel perhaps?). Or maybe that is why big companies (Google, IBM) are all hiring Mozilla developers....(or maybe they just believe the browser is an important platform. We all know Google is a company that does no evil). Ok enough conspiracy theory. Maybe KDE can re-fork the fork... Note: I don't have a personal vendetta against Apple. I have school friends, old co-workers, neighbors and ultimate pals who all work for apple (none in the browser division though). I even recommend Macs to friends and family and even almost bought a mini.... Thoughts?

All your bits are belong to us (errr bitkeeper)

2005-04-26T00:00:00-06:00

So a picure is worth a thousand words... I'll try and explain this in less than 1000 words. My brother and I were talking about the BitKeeper incident last week. He claimed that the revocation of the BK license to OSDL and that Linus' subsequent droppage of BK is prototypical for those who don't play fair with open source in the software industry. The OS community will move on. That seems pretty harsh but might not be too far off the mark. (When not using "free" software you are subject to the whims of the vendor, even if their proprietary software makes your job easier).

But enough waxing Stallman. For me it is interesting to observe the catalysts that drive open source adoption as well as those that drive features. For many open source projects, they seem to grow organically, and sometimes that growth happens in large spurts. I'd like to conjecture why (I have no real proof, so feel free to correct me if I'm wrong, this is only based on casual observation).

In this instance, at the point in time I've labeled as the "debacle point", there was no open source software that provided Linus with the functionality that he needed to serve as the filter for kernel patches. This vacuum caused by BK was the catalyst for rapid development of Git and it's adoption among kernel developers.

A few notes. I believe the adoption is aided by the fact that a respected leader such as Linus was the one who developed Git. The adoption graph only represents usage among the kernel developers (I don't know what will happen in general to the adoption of BK, it seems they ostracized their reference client, but maybe in their eyes any news is good news (er marketing) for the BK folks). Git may not ever have all the features of BK, but it will eventually be "good enough". I make no guarantee about the scale of my graphs (I'm just trying to illustrate a point). I was kind of hoping that some of the other nascent distributed SCM projects would step up to the plate (bazaar-ng), and there would be mass improvements as projects competed to be Linus' pet SCM. (was that under 1000 words???)

Choprock incident hits close to home

2005-04-25T00:00:00-06:00

The news of the two men found in Choprock canyon last week hit me really close to home. Not just because I'm from Utah, but also because I have spent a bit of time hiking in the canyons of Southern Utah during my college days. I could envision myself in their shoes. I had a similar feeling reading Aron Ralston's depiction of his entrapment in "Between a rock and a hard place".

I've never been to Choprock, but have visited a few canyons in the Escalante area. My old college roommate decribes the canyon and sums up my feelings pretty well: Choprock is supposedly deep, dark, narrow and prone to getting log-jammed. From what I have been able to glean from the news and folks on the canyoneering yahoo group, it looks like a big unstable log jam backed up a deep pool of water and these two guys got stuck, hypothermia set in, and they drowned. The initial report was that their bodies were found in 15 feet of water. That's a ton of cold cold water. Just typing out that last paragraph gave me butterflies

Marten Mickos keynote observations

2005-04-22T00:00:00-06:00

Last Wednesday I attended a very interesting keynote given by Marten Mickos entitled "Scale Out". Here's a few gems that I gleaned from it: He stated that the survivors of the dot com era built their infrastructure on top of open source (and mentioned Yahoo, Google and eBay as examples).

Standardization is a growth forumla for any industry. As compoments such as databases standardize it allows for growth in volume (those who couldn't afford a database now can). The surge of new buyers allows and enables those users to innovate and create new applications. He didn't really define what he meant by standardization (SQL Standard?). Big companies selling expensive software are afraid of standardization.

He made an analogy with the GPL and a Swedish term and idea. I didn't write down the term but the idea was that anyone in Sweden is free to use any empty land (or even land purchased by others, even if the owner has put up a fence). But when someone wants to do certain things (such as hunt, build a fire or house) on the land they must pay the owner. An interesting analogy for their dual licensing scheme.

Marten explained the ratio of users to contributers to paying customers (1000 to 10 to 1). He said they were his Marketing, Product and Sales department respectively, and that MySQL was perfectly happy with that ratio as it was allowing them to be profitable and grow. He then stated a law he called "Geoffrey Moores" law (the guy who wrote Crossing the Chasm) that every 18 months the users of opensource doubles. (I'm not sure I buy that one but it sounds nice).

Marten said there are two types of users, community users and customers. Community people are willing to spend time to save money. And customers will spend money to save time. MySQL supports and enables both of those users, and it is a fine line to provide extra value to those paying customers without causing the community to become disenfranchised. He said that Moore had also recently expounded (at OSBC?) on the 4 types of cultures: cultivation (Google), Competition (MS), Collaboration (OpenSource) and Control (IBM). I didn't really follow that point, if anyone can clear that up go ahead.

The 3 priorities for MySQL are 1) Performance 2) Reliability and 3) Ease of use. He indicated that this explained MySQLs position in the market (web dbs versus enterprise dbs, but that they were progressing in the enterprise arena). One of his final analogies was that MySQL was the JetBlue of databases. If you want to fly business class you can use DB2, Oracle or SQL Server, but most people don't need/have to fly business class. All in all it was pretty interesting, though I'm still not sure what scaling out is.

Trogdor comes in the night.... so leave the light on

2005-04-22T00:00:00-06:00

The past few nights our daughter Olivia has claimed that there are monsters in her room (too much Monsters Inc?). She has been sleeping with the lights on. When the lights are on, she is completely calm and has no fear, and finally relaxes and goes to sleep. Pretty easy to make an analogy to Open Source. There is comfort is being able to see what your code is doing if necessary.

Demoing SpikeSource at MySQLUC

2005-04-20T00:00:00-06:00

Today I was lucky enough to attend the MYSQL Users Conference for most of the day. The keynote by Marten Mickos which was very interesting (will blog on that later).

It's very interesting to see how far MySQL has come. (We used MySQL for my senior project at school. While pitching the VCs (hey it seemed like the cool thing to do during the bubble) they were concerned that we had used MySQL instead of "a real SQL database". To make the story short, we didn't get funding and the 5 other companies that were doing the same thing went under in less than a year.... Now MySQL is used all over the place. How far we've come in 5 years!)

During the afternoon we demoed SpikeSource's native component builds ( http://spikesource.com/spikewatch/index.jsp ), which is currently pulling from the head, building and testing Geronimo, Lucene, Nutch and MySQL. Most of the people I talked with were developers who understood the need for tested, integrated components and were very interested in our stack.

Ok, I'll try to blog my version of the Marten's keynote tommorrow...

Baypiggies meeting with Guido

2005-04-14T00:00:00-06:00

Tonight I went to the well-attended repeat of Guido's Pycon 2005 talk "The State of the Python Union". He started off by explaining his "beta [crunch] beard" and how he was now addressed as professor due to it. Then he quickly discussed the syntax of generators and how he copied the Java sytax.

He seemed happy that Infoworld is reporting that 14% of users in its survey used Python and that Python recieved the Jolt Productivity runner-up award (since the last time the recieved it was 2000, he noted that it should be a 5 year cycle ;)). As another sign of Python's maturity he discussed the first python vulnerability alert and the creation of a vulnerability mailing list.

He then spent a while discussing the future of the language, stating that Py3k was "a few years out" still (same status it had in 2000). On the next versions of Python, Guido would be focusing on the language syntax and libraries, though he stated that quickly moving 3rd party libraries (GUI and web toolkits) will remain out of the standard library since their release schedules conflict with the stable 1.5 year python cycles. Regarding language design he noted that for him a language being "more expressive was better than speed" noting his current suggestion that functional constructs be replaced by list comprehensions. He also stated that this was "doing the reader a favor".

The remaining time was spent discussing optional type checking (which he called "idiot checking") and interfaces (which big projects such as Zope and Twisted are already implementing in python). There was a bit of discussion and Guido seemed as uncertain as the audience regarding what to do here.

Guido is a humerous person and it's comforting to know that someone who is looking out for the reader of code is keeping the python language in check. It will be interested to see what happens static typing and interfaces.

Back in Google

2005-04-12T00:00:00-06:00

Sometime today I was put back in google's index

I have no clue what happened. Are they just toying with me?

Even More Competition Ubuntu and Debian...

2005-04-12T00:00:00-06:00

So a week ago I was able to install Hoary Hedgehog. What can I say? The install was simple, and it worked. Kubuntu worked as well. I had been itching to install debian for a while to try it out, yet as I'm a more bleeding edge type of guy, I knew I wouldn't want to use stable (yeah I know testing and experimental). (I was impressed with Ubuntu, but will be sticking with Gentoo for the moment.)

So there was a recent interview with Debian Guru Ian Murdock. Here's an interesting snippet: "If anything, Ubuntu's popularity is a net negative for Debian," Murdoch told internetnews.com. "It's diverged so far from Sarge that packages built for Ubuntu often don't work on Sarge. And given the momentum behind Ubuntu, more and more packages are being built like this. The result is a potential compatibility nightmare." Murdoch argues that if Ubuntu were truly compatible with Debian, all of the energy going into it could be directed at Sarge and toward getting it released, which is what would really benefit the Debian developer ecosystem as a whole. "I understand what the Ubuntu folks are trying to do, and they're doing lots of good work that will eventually find its way into Debian," Murdoch said. "But what we really need right now as a community is for Sarge to be released. "In that respect, Ubuntu's popularity is more harmful than helpful." Hmmm, my feelings were that Ubuntu was a positive for the linux community as a whole. It will be interesting to see what happens but I think Debian will be forced to streamline or this "fork" called Ubuntu may steal a lot of their userbase. It appears that Ubuntu is giving a lot of Debian users more of what they want....

Burninated by Google

2005-04-11T00:00:00-06:00

Hmmm, there seems to be a conspiracy brewing. One day I'm blogging about how I'm ranking pretty good for some searches in Google (considering that few read or link to my blog) and the next day Google no longer acknowledges my existence. The three of you that read my blog on Saturday probably think I'm crazy!

Soon after I blogged that entry, Google has removed my blog from their indexes! What did I do? Did the big brother bots of Google notice that I used the wording "Google bomb" and remove me as if they were the TSA and I was trying to carry some toenail clippers onto an airplane? So I sent the following email to Google:

Subject:What happened to my blog?

Dearest Google-folk- It appears that my blog ( http://panela .blog-city.com/ ) has been dropped from your index. This happened soon after I blogged about how my website was ranking high in google ( http://panela.blog-city.com/read/1189118.htm ). Would that entry have something to do with my removal from your indexes? I'm sure the few viewers (you could count them on your fingers) of my blog don't really care too much, but it is a shame since my blog does happen to be the only website detailing how to use the Rocbox mp3 device on Linux. Please let me know why google thinks that my blog no longer exists.

yours truly, Matt

In less than 5 minutes I recieved a canned response stating the following:

Thank you for your note. This is an automated reply to your inquiry about your site's inclusion in the Google search results. When webmasters write to us that their site has fallen out of our search results, we often find that it's still included.

To quickly determine whether your site is currently included in our index, perform a Google search for its entire URL. For example, a search for [ www.google.com ] returns the following result: http://www.google.com/search?hl=en&ie=UTF-8&q=www.google.com. If your site returns as a result, it's included. While we're always working to include more content in Google, sites can occasionally fall out of our search results. Our spiders regularly crawl the web to rebuild our index, but keeping tabs on eight billion pages is tough work, and they may miss a few. As for the sites they find, updates to the content of pages often result in ranking changes. Normal changes include, but aren't limited to, addition of new sites, changes in the ranking of existing sites, sites falling out of the index or getting dropped for particular keywords, and fluctuation between old and new webpage content.

Please be assured that these changes are automated. It is certainly our intent to represent the content of the internet fairly and accurately. Our crawlers aren't bullies; they don't pick on particular sites. Also, please note that Google's advertising programs are independent of the process by which we assemble the Google index. Participation in an advertising program does not affect inclusion or ranking in the Google search results. We understand that these changes can be confusing. If your site's well linked to others on the web, it's likely that we'll add it again during our next crawl. While we can't guarantee that any page will consistently appear in our index or appear with a particular rank, our Webmaster Guidelines, available at http://www.google.com/webmasters/guidelines.html, offer helpful tips for making your site crawler friendly. Following these recommendations should increase the likelihood that your site will show up consistently in our search results.

Regards, The Google Team

I followed their advice and my site does NOT exist as of 11:30pm in their index! Just for fun I'll list some of the searches that used to (as in they did 2 days ago) return my blog in the top 10 results:

Sticky clay at Fremont Older

2005-04-10T00:00:00-06:00

[]

I was able to go mountain biking for the first time in a few months yesterday. I've only been once since I had a nasty endo last fall at Soquel Demo Forest (picture above), leaving me attending physical therapy for a month.

Yesterday morning it appeared to be a beautiful day, even though it had rained a little on Friday. Everything was going fine until 3/4 of a mile up the trail turned into really thick mud. This isn't the wet kind of mud that you can splash through, get dirty and keep riding. It was more like wet clay, thick enough and sticky enough to build up around my front shock and seat and chainstay. It kept building up until my tires refused to turn. Dragging my bike I turned back. I ended up washing my bike and that took me longer than my short ride. The only consolation was that I was back in time to go play pickup ultimate. Hopefully the weather stays dry so I can get back to Soquel and Skeggs.

Google bombing with Dick Proenneke

2005-04-09T00:00:00-06:00

[]

An interesting observation. One of the cool features of blog city is that you can view statistics for your blog, which includes websites that refer to your blog. Interestingly enough if you search for "Dick Proenneke" my blog appears on page 3. In the UK it is the first entry for Dick! What would cause it to rise so far up? Very few people view my blog and no one links to it. I didn't know that being an SEO was so easy. ;)

Architecture of Participation, Wikipedia and Dguitar

2005-04-08T00:00:00-06:00

It's pretty amazing how the internet enables people to work together. Tim O'Reilly has dubbed this the Architecture of Participation. Just the other day I made an entry in wikipedia. Wikipedia has this really cool feature where you can see all of the edits to a page and do a diff against them. Well within minutes of me creating an entry (which was easy to do and enabled common folk such as me to participate) another person had edited my entry! Pretty amazing.

Just today, I witnessed the AoP again. A co-worker happens to play guitar and was telling me about some software he used to display tablature. I don't profess to be a good guitarist at all, but it is fun to try and make music (my wife and the few others who have heard me can vouch that I have no rhythm).

I basically taught myself to play some chords by getting a Neil Young easy guitar book. (It really was easy to play the simple stuff). But I really haven't progressed much beyond that. I've got pages of print outs of OLGA guitar tabs (which also was created by an AoP), but for a man with no rhythm it is hard to play just by that, what I really needed was a program that would play the music for me and show me what to play at the same time.

Amazingly enough there is a program called Guitar Pro that does just that. Sadly enough it is Windows only (so I can't run it unless I do it under wine...). The cool thing about this program is that they have a website called mysongbook.com <http://mysongbook.com/>__ that has tons of tablature in gp3 and gp4 format. (Even better than the piecemeal chords I have from Olga and the ocassional nice tab). End users can create tablature and submit it here. There appears to be quite a collection.

So after less than a minute on google I find dguitar, a nascent effort to support gp3 and gp4 tabs under the java platform. The project is less than 5 months old, and there appears to be collaboration on their forums. The latest dev version works great on my machine, the only feature that would be really nice (for someone like me who isn't able to play super fast) is to be able to set the tempo, but I'm sure that will come. Great work guys!

I don't think dguitar would even exist unless there were tabs in the gp3 and gp4 format (which appears to have enough benifit over traditional tabs and chords that warrants people writing an app to support it).

(Now if I could only replace those nasty swing file choosers with native kde or gnome, every time I click to open a folder it thinks I'm trying to rename it...)

The Long Tail of Boxes, Jeans and Trogdor

2005-04-06T00:00:00-06:00

Above is a graph illustrating the much blogged about long tail. It starts steep on the left side, continues through burnination and tapers off after a while to a little point.

I was talking with an acquaintance the other day and he was was telling me of a box business he was helping to start. Their business catered to small businesses and individuals who needed small amounts of custom sized corrugated cardboard (boxes). One of their clients was a bowling ball manufacturer. Large box companies wouldn't give anyone who ordered such small quantities of boxes the time of day, yet this man was convinced their company could satisfy the needs of many companies ordering odd size boxes and be profitable.

Later my mom was lamenting how difficult it was to order womens pants with long legs. I feel for her. It is very hard for me to find pants with a 36 inch inseam and a waist smaller than 34. So again we come back to the long tail. There exists people with needs that are not met by the majority. Who will supply these needs? Yes, I could order pants online, yet I'd much rather try them on before purchasing to know if they fit or not.

So can brick and mortar stores participate in this long tail phenomenon or will they get burninated along the way? Really, I'd like to know....

More competition, Ruby on Rails, Java and Python...

2005-04-05T00:00:00-06:00

More thoughts on competition... Justin Gehtland's blog where he mentions his delvings into Ruby on Rails was slashdotted today.

Here is a Java author (he wrote a book about the Spring Framework), jumping ship and advocating the use of Ruby for web development. I'm not really a Ruby person, but I keep hearing mentions of RoR among the python community. Python folk have been waiting for a "killer app" in python for a while that will validate python among the larger IT community (some claim Zope or possibly Twisted is this app...).

Now RoR comes out of nowhere and in less than 6 months everyone is talking about how it is the greatest thing since sliced bread. (To be fair to python, python probably has all the parts of RoR but not in one coherent package, plus there are many different web frameworks for python, so it is hard to know which is the one true framework.) Even Bruce Perens is working on a slashdot clone in RoR for his technocrat.net site.

Justin reports that the source code for his RoR app alone is smaller than the size of Java's config files for the same app! He also claims that his Ruby app is as performant as the Java version. I'd like to hear about scalability as well.

So what does this have to do with competition?

I'm not trying to fan any flames here, I'm sure that there are holes in RoR and that we aren't really comparing apples to apples here. Language zealots are being kept on their toes because features/capabilities/productivity enhancements are appearing in other languages. Java and python frameworks will need to absorb the features of RoR to keep early adopters/technologists from jumping ship or improve their marketing.

Burninating before burninating was cool....

2005-04-04T00:00:00-06:00

Coming soon...

"P-Language Report"

2005-03-28T06:45:00-07:00

Note - This is a migration of content from my old blog.

It looks like the report mentioned by Guido is now available. It meantioned that {"programmers are 5 times more productive when using "P" languages"}

Here is a writeup on the report. On a side note, I've been playing around with PHP a little bit. Some things that are bothering are references (in version 4, 5 fixes this) and classes which seem like an afterthought.

No Go for the Rocbox

2005-03-22T00:00:00-07:00

So I'm going to take the Rocbox back. I should think twice before buying a first generation item from a vendor (even if the price is right).

There were a few things that I liked about it:

The feel of the buttons on the front (they click well like an old hp calculator)
The side scroll navigation actually works pretty well for one hand navigation (though it takes too many clicks...)
The thing feels pretty solid

Now to the bad things:

Battery life for me is about 5 hours. This is unacceptable. I recharge the thing every night and listen to it during work (turned off during lunch), and I only get about 5-6 hours out of the thing.
No resume! I've got potentially 20 gigs of music, and every time I turn the thing off it forgets where I was (don't want to leave it on with the way it goes through the batteries). This is very annoying.
Syncing is more of a hassle and I've heard complaints about playlists not working (I never implemented that in python....)

What to do? I'm actually very intrigued by the iriver h320 ( http://www.iriveramerica.com/prod/hd/h320.aspx ), it appears to have most the features I want (though I'm not sure it will have nice buttons like the rocbox):

>= 20 gigs
Decent battery life
FM radio
microphone
Drag and drop syncing
apple ipod price tag

Well all but the last one are very good. The reviews actually seem pretty good too. Anyone want to trade one for a slightly used rocbox?

Open Source and Competition

2005-03-22T00:00:00-07:00

People complain that their favorite app isn't ported to Linux, but when such an app is ported they complain. With the recent release of Nero cd burning software for linux, I've read a few gripes like why Nero used gtk+, is it free (as in gratis)?, why even release it since k3b is so good, etc. Sure it is easy to find something to complain about, but Neros release is competition for open source equivalents, and I believe that most competition is good for open source.

I just installed KDE 3.4 on Gentoo a few days ago. This quality of this software is leaps and bounds above what I got when I first installed Linux in 1998. Why? KDE and Gnome continue to leap frog each other in features, improvements and applications. They push the buttons of developers from the other camp and the software from both sides sees improvements. With only one desktop, this innovation would be a lot slower. (I actually use a lot of gnome software as well, gnomemeeting, gimp, inkscape, but realize that "purists" want equivalents in kde thus improving apps on the kde side of things).

Gentoo is a similar story (I'm not saying the gentoo is the perfect solution for everyone), when I first installed Linux (using rpm-based distros), I would have to re-install every 3-6 months to get the latest software. (Trying to update an install would usually break it). Been using Gentoo for 2 years now, and the days of reinstalling appear to be over (I'm sure Debian people can say similar things). (Yes, I do have to wait for compiles, but with the new breakup of kde packages that time has shrunk significantly). Debian and the rpm based distros didn't provide Daniel Robbins with a system that felt good to him, so he went out and made his own. And I think this competition of new distros has made Redhat (or Fedora Core or insert distro of the week here) better. I'm using FC3 at work and Yum and synaptic are definitely an improvement over straight rpm. An improvement that wouldn't have come along as soon without pressure from distributions such as debian and gentoo.

An added benefit of open source competition is that it allows projects to cross pollinate each other not only at a high feature to feature level, but at a source code level as well. The transparent nature of open source can allow for frameworks and libraries to be used and improved by both camps. Freedesktop.org is an excellent example.

These are two simple example, but I still believe competition is good for open source, even if it comes from commercial outsiders. Consumers love competition (it drives down price and/or tends to improve quality). Instead of berating Nero for releasing a Linux version of their software, let's welcome it and see what we can learn from it.... I'm still waiting for DVD mastering software that is as easy to use as k3b is....

Adding Music to the Rocbox with Linux

2005-03-15T00:00:00-07:00

Ok, if anyone is reading this, I thought I might share some investigative work. So after a quick attempt to copy music over to the rocbox failed, I started searching on google for any help. Nothing turned up.... So I looked at the db files on the rocbox and googled for them. Nothing.... So I zipped up the contents of the db files and emailed them to my brother (he's smart and works on low level stuff, smart cards and the like), so I figured he'd be able to give me a head start. Nothing.... So I googled some more and finally came upon this link which explains how data is stored on a Philips HDD 100/120 Audio Jukebox. I can't even remember what I searched for, but it turns out that the file structure is very similar (just the prefix of the files were changed (db5000 -> pp5000). More googling on that led me to this which is a python solution for the Philips device (even better). After some quick hacking on that file I was able to upload most of my music to the device. (If you want this let me know and I'll get it to you, most of my issues were with character encoding, which I think I've worked out). Now I need to be fair to my brother. I found those links before he had time to answer me, but he was amazed that there is even a need for the db files. "Why not just have the hardware search for new files and create it's own db?", he asked. I replied that with 20 gigs of data you don't want to scan the whole thing even time you turn it on.... Well after transferring about 5 gigs of music and turning the device on play mode, it said "updating database" for about 10 minutes.... So that seems like the worst of both worlds. Oh well. (I'm thinking about extending the script so it works with amarok. Anyone else interested?? )

Alone in the Wilderness (Dick Proenneke)

2005-03-15T00:00:00-07:00

Just finished watching an interesting autobiography of Dick Proenneke on public television. He wanted to spend a year alone in Alaska (at the age of 51), living off the land and hand tools (he made a cabin). His cabin was amazing. He ended up living there for 30 years. Felt very similar to reading Desert Solitaire by Edward Abbey. Except Dick was in Alaska and Ed in Utah. http://www.nps.gov/lacl/scrapbook/proeneke.htm

Using the Rocbox with Linux (as a storage device)

2005-03-12T00:00:00-07:00

I recently bought a rocbox (http://www.roc-digital.com/main.html) for a good price, thinking I would be able to use it as a portable drive as well as an mp3 player. (I got it for about half the price of a 20 gig ipod), so I won't complain about the price. One brother and my sister both have ipods so I've been able to briefly play around with them, and must admit they are very nice. The usability on the ipod is superb.

So I'll report first off that the Rocbox does not compare with the ipod with respect to ease of use. The jog dial on the side is a nice idea, but it feels a little flimsy and seems to me that you have to push it too far when you want to use it as a buttton. Also the fonts and graphics on the screen are just not very nice to look at, I'm not sure if it is the screen resolution or what, but the columnar arrangement doesn't work well with me (I like how the ipod arranges its content in rows).

Anyway enough of a usability review... how do I use the thing with linux?

I plugged it in and typed dmesg, but I couldn't see that the device was detected. I knew that I had read somewhere that it could be used as a mass storage device, so I was wondering what was going wrong. After a few minutes I figured out I need to turn the rocbox on for it to be detected. Lo and behold it was detected as a mass storage device (after turning it on)! I added the following line to my /etc/udev/rules.d/10-local.rules (I'm running gentoo with a 2.6 udev kernel)

BUS="usb", SYSFS{product}="ROC-Digital rocbox", KERNEL="sd?1", NAME="%k", SYMLINK="rocbox"

With that I could mount the device to a mount point I added in /etc/fstab

/dev/rocbox /mnt/rocbox auto noauto,user 0 0

There was an existing directory structure on the device, I immediately deleted the hiphop files and transferred over some obscure Silvio Rodriguez music I had lying around. After disconnecting, the device still claimed to have the pre-loaded hip hop music and none of Silvio.... What happened??? I'll write that up later.

"Guido Van Rossum: Building an Open Source Project"

2005-02-19T23:08:00-07:00

Two nights ago I was able to attend a talked titled "Python: Building an Open Source Project and Community" given by the creator of Python himself at Xerox Parc.

Guido is a funny man (the name Python is from the comedy troupe "Monte Python" — not the animal Python) and shared his experiences and the reasons behind the Python language.

I'll share a few of the things that were interesting to me:

Guido currently works (since 2003) at a "stealth mode" startup Elemental Security, that is working on policy compliance. The code consists of java and Python. He wrote 20k of the 30k lines of Python code.
Encapsulation is poor in Python. There are no "private" variables. But this design makes it trivial to create a profile, debugger and provide introspection.
Language Design: He wanted Python to be flexible and extensible. I believe that this is one of the key areas that has led to the community around Python. It is easy to write modules for the language so gui, db, web people can all create modules for their own needs.
Someone asked why Python programs can run for days on end while other languages (specifically Java in this case) run out of memory? Guido traced this back the the simple design of the garbage collector which is doing reference counting (plus cycle/loop detection), while other runtimes implement more difficult memory management schemes which lead to greater overhead (in startup time) and more complicated design.
Of the "P" languages (Perl, Python, Php) he said the Php is the most successful because it delivers a specific need to a niche. But he said that the design of the language itself is poor. Version 5.0 and newer versions should address some issues.
According to the Burton Group, programmers are "5 times more productive" when using "P languages". Also, the Burton Group said that Python is the "preferred" p language for application development. Guido attributed the productivity increase (for Python at least) to the fact that Python code is much shorter (he said around 5 times) than C, C++, and Java because there is much less syntatic sugar (no type declarations). He also said that he feels a programmer’s productivity is bounded more by lines of code than functionality of code. Thus a 500 line Python program is more maintainable than a 2500 line c program that does the same thing. (Guido also admitted that there are places where Python is not appropriate, writing a mp3 decoder is an example). He also said that most programmers are not fit to do memory management (C/C++).
When asked about static type checking, he basically said that it was a crutch. The "benefit" of static type checking was that the compiler would warn you of type mismatches. Guido claims that these errors will be caught be unit testing anyway. (and since java developers are supposed to be writing unit tests the value of static typing erodes) The harder type of bugs (value errors) will not be caught by the compiler. See Bruce Eckel's take on this , the author of Thinking in Java.
Python is object oriented, procedural and functional. Guido said he didn't really know what functional means, but Paul Graham does, and he says that "Python is most like lisp", so that makes it a good langauge. See Paul Graham’s essay on Python where he makes other interesting claims about Python and google.
Licensing: Guido says wished he could keep the Python license to 2 paragraphs, but the lawyers said that "can't be a license". He feels good that the license is OSI approved. He says that half-free licenses won't work (free to universities but not for commercial). He also says that opensourcing abandonware won't make it popular ("if your software isn't used before it is opensourced, it won't be afterwards"). He also noted that Python's license was "non-gpl" so people would feel comfortable using Python in their environment.
Debugging: Guido uses print statements for 90% of his debugging and a debugger for the times when his brain stops working or he has to debug some wierd code that someone else wrote.