One Big Fluke

Headline I never imagined when I started working on open standards at Google

Thu, 23 May 2013 23:41:36 -0700

“Google Abandons Open Standards for Instant Messaging”

Sigh.
from One Big Fluke http://bit.ly/16dySn3

Know you're right: Always triangulate conclusions

Tue, 21 May 2013 13:46:09 -0700

I went to AAPOR for the first time this year, a conference covering public opinion polling and survey research. It’s hardcore. Using “data” as a singular noun there is gauche. My goal was to see the panel session on Pew’s use of Google Surveys. I had a great time and would go again.

A highlight was when I used triangulation to correct the data in an economist’s presentation (never done that before!). It made me realize that beyond academia, entrepreneurs and startups should also be using triangulated research to validate their product plans and business models.

What is triangulation?

Triangulation is when you ask the same question many different ways and compare the results. You’ll see agreement or disagreement between the questions. If they don’t agree, something is happening that you don’t understand. This lets you self-validate or corroborate your findings. Think of it like running an A/B test on survey question correctness, except that you want zero separation. It’s similar to a fundamental part of the scientific method.

To show what I mean, I ran four different survey questions about cat ownership in the US. Here are the results (after some simple math):

#	Question	Cat ownership	Dog ownership	Pet ownership
1.	What kind of pets do you have in your household?	23.9% (+2.2/-2.0)	41.0% (+2.4/-2.4)	~52%
2.	How many cats do you have in your household?	28.2% (+2.3/-2.2)	-	>= ~28%
3.	Do you have one or more cats in your household?	24.0% (+2.1/-2.0)	-	>= ~24%
4.	Are cats or dogs not present in your household? Or do you have both types of pet?	30.5% (+2.4/-2.3)	44.0% (+2.5/-2.5)	~58%

The results converge on the same numbers (cat ownership between 22% and 28%) and agree with each other. Most differences are within the margin of error. The min/max span is 4 percentage points. The numbers also agree with data from the humane society and the AVMA sourcebook. I’m confident I know how many people online have cats.

What happens without triangulation

At AAPOR, one of the researchers from NORC / University Chicago presented test results on how representative Google Surveys are. Their original, less accurate finding was that Google Survey data does not closely agree with benchmarks for telephone ownership. We ran a follow-up survey to find out why. The problem was modal bias: Asking a question over the phone introduces errors that are different than the errors from a microsurvey.

By tweaking the question slightly we were able to reproduce the Pew Internet data within 3 percentage points (our results are here in Q1/2; Q3/4 demonstrate the modal bias).

The NORC folks were happy to hear our data was better than they thought. Had they triangulated their results themselves, they would have seen disagreement and known that something else (modal bias) was happening.

Why not triangulate?

Surprisingly, nobody I asked about triangulation at AAPOR had employed it in their own research. Maybe I missed somebody, but it makes sense:

Most polling that exists today is extremely rigorous and proven.
But it’s slipping away because:
- It’s illegal to robo-call cellphones.
- Phone response rates are very low.
- Your current location and cell phone area code are often unrelated.
This makes traditional market research and opinion polling expensive and introduces bias.

So it’s plausible that traditional researchers don’t triangulate because they can’t afford to. And why would they triangulate if the existing measures and techniques work well? The problem is when old measures are applied to new situations, like the NORC example above.

Now it’s easy

With new methods it’s cheap to do triangulation. I’ve seen startups triangulating decisions using Google Surveys and the results are great. I presented one such case in London, recently. Anyone can do it.

So: Whenever you make an important decision about a product, business, or research you should triangulate the data used in your conclusion. Try multiple approaches and find agreement between many measures of the same idea. This will give you confidence in your conclusions. It will provide defense against detractors. It will bring consensus to your team.

And you’ll know that you’re right.
from One Big Fluke http://bit.ly/10jydfc

Piglet and Pooh Bear

Sat, 18 May 2013 12:56:34 -0700

“When you wake up in the morning, Pooh,” said Piglet at last, “what’s the first thing you say to yourself?”

“What’s for breakfast?” said Pooh. “What do you say, Piglet?”

“I say, I wonder what’s going to happen exciting today?” said Piglet.

Pooh nodded thoughtfully. “It’s the same thing,” he said.

from One Big Fluke http://bit.ly/12CZfxY

Fixing Security using Continuous Deployment

Wed, 15 May 2013 10:26:45 -0700

I enjoyed this slide deck from Nick Galbreath, especially slide 25, which states the following hypothesis:

It is impossible to simulate the production environment in development, either due to operational differences or data differences.
No amount of QA or Security Testing can prove you don’t have bugs, vulnerabilities, or cause severe operational problems.
You have bugs and vulnerabilities, right now, in your application.

And the conclusion is that the only solution is continuous deployment. Indeed! I’m happy to see this viewpoint taking hold.

Here are the slides:

from One Big Fluke http://bit.ly/10GL8UZ

The Isley Brothers – Footsteps In The Dark - Part 1 & 2 from One Big Fluke http://bit.ly/16whwAU

Mon, 13 May 2013 23:56:28 -0700

The Isley Brothers – Footsteps In The Dark - Part 1 & 2
from One Big Fluke http://bit.ly/16whwAU

Video: Cohort Analysis talk at Google Ventures Startup Lab

Mon, 13 May 2013 21:41:30 -0700

The Google Ventures Startup Lab posted the video of my talk about Cohort Analysis. The slides are here.

Enjoy!
from One Big Fluke http://bit.ly/15HOoaE

First world pants

Sun, 12 May 2013 13:11:35 -0700

I’ve talked about my preference for first world goods previously. The gist of it is, I want to buy things made by people who have the same rights and freedoms that I do. I’ve managed to find great first world sneakers, shoes, tees, hoodies, shirts, and more. The missing bit has been a decent pair of jeans.

There’s a place in SF called Self Edge that, for no real reason, I always assumed was snooty and bullshit. They also have a store in NYC, and I stopped in with a friend on a recent trip, putting my skepticism aside. That day, Kiya Babzani, the store’s founder, happened to be in the NYC shop. We talked for a while and it turns out he’s a rad guy. It made me realize that Self Edge isn’t snooty, it’s ~~border-line~~ Otaku. What they sell is mostly made in Japan, where the car/motorcycle/rockabilly subculture has co-opted classic American manufacturing and brought it back to life.

Which brings us to the pair of pants I bought. Check out the well-executed, overly descriptive, not-quite-but-still-is-Engrish on this tag (e.g., “vintage sleek”):

Anyways, these pants are great. They’re sturdy enough that they identify them by weight more than anything. I figure they’ll last five years, probably more, justifying the cost to me (it helps that the Yen is super weak right now). Contrary to popular belief, you’re supposed to wash them frequently.

If you’re still skeptical, try reading this two part interview with the founder. I enjoy things that people are passionate about, like bikes, to the point of being obsessive. The Flat Head and Self Edge pass the test. I stand corrected. And now I have a nice pair of first world pants.
from One Big Fluke http://bit.ly/14fPk0W

I actually sit down and open my postal mail maybe four times a year. Wish I could stop it. from One...

Sat, 11 May 2013 17:41:34 -0700

I actually sit down and open my postal mail maybe four times a year. Wish I could stop it.
from One Big Fluke http://bit.ly/YBOHye

Growing Tomatoes

Sat, 11 May 2013 11:43:46 -0700

The past few summers I’ve attempted the near-impossible in San Francisco: Growing tomatoes. Every day it’s some combination of cold, foggy, and windy. But I’ve managed.

What’s surprising about growing tomatoes is how something so small can grow so big. I start with a tiny sapling in a thin plastic pot. I move it to a larger pot and then water it every couple of days. By the end of a week its mass has doubled. At a month it’s over a foot high and blossoming. After two months the first tomatoes are growing fast and the plant is chest-high.

Each day I water the plant it looks the same as the day before. I can’t notice daily changes because the differences are subtle. I went away for a week and when I got back the plant looked enormous compared to before. When I hadn’t seen the incremental changes, the plant’s growth was astonishing. Growing tomatoes has made me see the value solely in time passing.

How would I be if I spent a tiny bit of time cultivating myself every day? It wouldn’t seem like much to me, since I’d witness the small differences. It’d be hard to stick with it. But after two months or a year, I may look back and realize how far I’ve come. I think this is how I developed as a programmer. I wonder how else I could improve this way.
from One Big Fluke http://bit.ly/10q1q3E

Bad marketing email. Not even trying to be experimental about it. No call to action. Too much...

Fri, 10 May 2013 17:11:34 -0700

Bad marketing email. Not even trying to be experimental about it. No call to action. Too much text.

from One Big Fluke http://bit.ly/199Urkn

From this article: “The people you want to hire aren’t applying and interviewing, they’re...

Fri, 10 May 2013 12:56:26 -0700

From this article: “The people you want to hire aren’t applying and interviewing, they’re running their own companies.” Or at Google, Facebook, Twitter, etc.
from One Big Fluke http://bit.ly/11YUrkB

All useful code must be rewritten at least twice. from One Big Fluke http://bit.ly/ZLP1rK

Thu, 09 May 2013 17:56:44 -0700

All useful code must be rewritten at least twice.
from One Big Fluke http://bit.ly/ZLP1rK

“Today we can officially confirm the release date for the Mac version of SimCity is June...

Wed, 10 Apr 2013 09:43:29 -0700

“Today we can officially confirm the release date for the Mac version of SimCity is June 11”. Sweet.
from One Big Fluke http://bit.ly/10SM1uQ

For the uninitiated

Mon, 08 Apr 2013 23:43:31 -0700

John Carmack’s magical sqrt (). Came up in conversation again tonight. It’s amazing!

from One Big Fluke http://bit.ly/10IRjsj

First game of the season

Mon, 08 Apr 2013 22:26:20 -0700

from One Big Fluke http://bit.ly/150O6tY

The feeling when the domain you want for your project is available. from One Big Fluke...

Sun, 07 Apr 2013 13:11:22 -0700

The feeling when the domain you want for your project is available.
from One Big Fluke http://bit.ly/143Bl2d

I dislike dubstep because it’s so desperate for attention. from One Big Fluke...

Sun, 07 Apr 2013 00:11:24 -0700

I dislike dubstep because it’s so desperate for attention.
from One Big Fluke http://bit.ly/Z121Mq

Woah, end of an era.subprocess.CalledProcessError: Command '['java', '-version']' returned non-zero...

Sun, 07 Apr 2013 00:11:22 -0700

Woah, end of an era.

subprocess.CalledProcessError: Command '['java', '-version']' returned non-zero exit status 1

I don’t even have it installed.
from One Big Fluke http://bit.ly/Z121Mm

Tech bubble: Your problems aren't everyone's problems

Sat, 06 Apr 2013 15:43:21 -0700

The other day I saw a tweet go by that made me think twice:

The idea that this issue is big enough to win the popular vote for President seemed questionable.

I wondered:

Who would want to use electronics during takeoff/landing?
How many people actually own a smartphone or tablet at all?
And how many people fly on an airplane regularly anyways?

If you are reading this, chances are you fall in these categories and so do your peers. But the real answers are surprising.

The data

Baseline for perspective: 81% of US adults use the Internet. 67% of them are on Facebook. (Pew)

87% of Americans have a cellphone, but only 45% have a smartphone. 31% have a tablet. (Pew)

The kicker: 56% of US Internet users have not been on an airplane this past year. 18% never have. (Google Surveys)

Let that sink in

Only 43% of the US Internet population has been on an airplane in the past 12 months
More people have used Facebook than have flown on an airplane this year
More people own a smartphone than have flown on an airplane this year

So don’t assume that everyone’s experience is just like yours. As we see in the data, outside of your circle the world is very different.

I don’t think we should feel ashamed for living in a bubble and having privileges. Products like Dopplr and TripIt are very valuable to their niche of frequent travelers. As long as we acknowledge how lucky we are, it’s fine. The problem is when people don’t realize they’re a niche and assume everyone thinks like they do.

In the end

Perhaps Ben’s tweet was deadpan, a hilarious critique of Senator Claire McCaskill, Democrat of Missouri, who really should be focusing on the welfare of the majority of Americans, not the privileged few.

(Discuss on YC HN)
from One Big Fluke http://bit.ly/10qFpF0

HN is so far gone now that someone felt compelled to put “Time for a technical article”...

Fri, 05 Apr 2013 09:43:41 -0700

HN is so far gone now that someone felt compelled to put “Time for a technical article” in their submission title.
from One Big Fluke http://bit.ly/14JMODn