Don’t mess with the Dog

Apparently Dog the Bounty Hunter — Mark Driscoll’s “Manly Missionary Award” winner — was arrested by U.S. marshalls for extradition to Mexico because he arrested a rich perv who was hiding out in Mexico and brought him back to the States.  I realize that Mexico is probably within its rights under our mutual extradition treaties to demand this, but it’s symptomatic of a sick system — one that plays games with “laws” instead of honoring and pursuing justice.

Techie Wannabes

So, for my day job I work on a web site.

There’s another web site where some of the more, uh, “avid” users of our site congregate and comment on us and how we’re doing.  One of these “fans” made a snide remark just yesterday about us needing to do some “basic testing” on our web site before we put it out there.

Dude, you have no idea.  We have three - count them, three - separate automated test suites, each with hundreds of individual test cases, all of which get run against our software before a new version of our site launches.  We spend several weeks bug bashing, load testing, and vetting release candidates before we launch them to the public.  We take the release process - including testing - VERY seriously, because we also wear the pagers that go off when someone doesn’t like something they see on the site.

Our web site isn’t just a bunch of perl scripts and a MySQL database.  It’s composed of more than twenty interlocking components, databases, and service, each of which runs on a fleet of machines, some of which are owned by other teams.  Those systems can fail with no notice to us (as hardware tends to do), or change in sometimes subtle ways with little or no notice to us (as software owned by other teams tends to do).  Anticipating all of the corner cases — especially the ones where every conceivable combination of services can die, brown out, studder, or generally go flaky — is hard.  Anticipating and hardening ourselves against the errors that those other teams will make is even harder (read: “impossible”).

So chill.  We’re not a bunch of amateurs, and it will get fixed, but it takes time.  And snarky comments on forums that say “this is broken” but don’t tell us who you are, when it happened, what you were doing, and what happened are worse than useless, because they cause managers to assign us to dead-end rabbit hole investigations instead of actually working to improve the stability of the site and the quality of the codebase.