Headline I never imagined when I started working on open standards at Google
“Google Abandons Open Standards for Instant Messaging”
Sigh.
from One Big Fluke http://bit.ly/16dySn3
Know you’re right: Always triangulate conclusions
I went to AAPOR for the first time this year, a conference covering public opinion polling and survey research. It’s hardcore. Using “data” as a singular noun there is gauche. My goal was to see the panel session on Pew’s use of Google Surveys. I had a great time and would go again.
A highlight was when I used triangulation to correct the data in an economist’s presentation (never done that before!). It made me realize that beyond academia, entrepreneurs and startups should also be using triangulated research to validate their product plans and business models.
What is triangulation?
Triangulation is when you ask the same question many different ways and compare the results. You’ll see agreement or disagreement between the questions. If they don’t agree, something is happening that you don’t understand. This lets you self-validate or corroborate your findings. Think of it like running an A/B test on survey question correctness, except that you want zero separation. It’s similar to a fundamental part of the scientific method.
To show what I mean, I ran four different survey questions about cat ownership in the US. Here are the results (after some simple math):
| # | Question | Cat ownership | Dog ownership | Pet ownership |
| 1. | What kind of pets do you have in your household? | 23.9% (+2.2/-2.0) | 41.0% (+2.4/-2.4) | ~52% |
| 2. | How many cats do you have in your household? | 28.2% (+2.3/-2.2) | - | >= ~28% |
| 3. | Do you have one or more cats in your household? | 24.0% (+2.1/-2.0) | - | >= ~24% |
| 4. | Are cats or dogs not present in your household? Or do you have both types of pet? | 30.5% (+2.4/-2.3) | 44.0% (+2.5/-2.5) | ~58% |
The results converge on the same numbers (cat ownership between 22% and 28%) and agree with each other. Most differences are within the margin of error. The min/max span is 4 percentage points. The numbers also agree with data from the humane society and the AVMA sourcebook. I’m confident I know how many people online have cats.
What happens without triangulation
At AAPOR, one of the researchers from NORC / University Chicago presented test results on how representative Google Surveys are. Their original, less accurate finding was that Google Survey data does not closely agree with benchmarks for telephone ownership. We ran a follow-up survey to find out why. The problem was modal bias: Asking a question over the phone introduces errors that are different than the errors from a microsurvey.
By tweaking the question slightly we were able to reproduce the Pew Internet data within 3 percentage points (our results are here in Q1/2; Q3/4 demonstrate the modal bias).
The NORC folks were happy to hear our data was better than they thought. Had they triangulated their results themselves, they would have seen disagreement and known that something else (modal bias) was happening.
Why not triangulate?
Surprisingly, nobody I asked about triangulation at AAPOR had employed it in their own research. Maybe I missed somebody, but it makes sense:
- Most polling that exists today is extremely rigorous and proven.
- But it’s slipping away because:
- It’s illegal to robo-call cellphones.
- Phone response rates are very low.
- Your current location and cell phone area code are often unrelated.
- This makes traditional market research and opinion polling expensive and introduces bias.
Now it’s easy
With new methods it’s cheap to do triangulation. I’ve seen startups triangulating decisions using Google Surveys and the results are great. I presented one such case in London, recently. Anyone can do it.
So: Whenever you make an important decision about a product, business, or research you should triangulate the data used in your conclusion. Try multiple approaches and find agreement between many measures of the same idea. This will give you confidence in your conclusions. It will provide defense against detractors. It will bring consensus to your team.
And you’ll know that you’re right.
from One Big Fluke http://bit.ly/10jydfc
Piglet and Pooh Bear
“When you wake up in the morning, Pooh,” said Piglet at last, “what’s the first thing you say to yourself?”
“What’s for breakfast?” said Pooh. “What do you say, Piglet?”
“I say, I wonder what’s going to happen exciting today?” said Piglet.
Pooh nodded thoughtfully. “It’s the same thing,” he said.
from One Big Fluke http://bit.ly/12CZfxY
Fixing Security using Continuous Deployment
I enjoyed this slide deck from Nick Galbreath, especially slide 25, which states the following hypothesis:
- It is impossible to simulate the production environment in development, either due to operational differences or data differences.
- No amount of QA or Security Testing can prove you don’t have bugs, vulnerabilities, or cause severe operational problems.
- You have bugs and vulnerabilities, right now, in your application.
And the conclusion is that the only solution is continuous deployment. Indeed! I’m happy to see this viewpoint taking hold.
Here are the slides:
from One Big Fluke http://bit.ly/10GL8UZ
The Isley Brothers – Footsteps In The Dark - Part 1 & 2
from One Big Fluke http://bit.ly/16whwAU
Video: Cohort Analysis talk at Google Ventures Startup Lab
The Google Ventures Startup Lab posted the video of my talk about Cohort Analysis. The slides are here.
Enjoy!
from One Big Fluke http://bit.ly/15HOoaE
First world pants
I’ve talked about my preference for first world goods previously. The gist of it is, I want to buy things made by people who have the same rights and freedoms that I do. I’ve managed to find great first world sneakers, shoes, tees, hoodies, shirts, and more. The missing bit has been a decent pair of jeans.
There’s a place in SF called Self Edge that, for no real reason, I always assumed was snooty and bullshit. They also have a store in NYC, and I stopped in with a friend on a recent trip, putting my skepticism aside. That day, Kiya Babzani, the store’s founder, happened to be in the NYC shop. We talked for a while and it turns out he’s a rad guy. It made me realize that Self Edge isn’t snooty, it’s border-line Otaku. What they sell is mostly made in Japan, where the car/motorcycle/rockabilly subculture has co-opted classic American manufacturing and brought it back to life.
Which brings us to the pair of pants I bought. Check out the well-executed, overly descriptive, not-quite-but-still-is-Engrish on this tag (e.g., “vintage sleek”):
Anyways, these pants are great. They’re sturdy enough that they identify them by weight more than anything. I figure they’ll last five years, probably more, justifying the cost to me (it helps that the Yen is super weak right now). Contrary to popular belief, you’re supposed to wash them frequently.
If you’re still skeptical, try reading this two part interview with the founder. I enjoy things that people are passionate about, like bikes, to the point of being obsessive. The Flat Head and Self Edge pass the test. I stand corrected. And now I have a nice pair of first world pants.
from One Big Fluke http://bit.ly/14fPk0W
I actually sit down and open my postal mail maybe four times a year. Wish I could stop it.
from One Big Fluke http://bit.ly/YBOHye
Growing Tomatoes
The past few summers I’ve attempted the near-impossible in San Francisco: Growing tomatoes. Every day it’s some combination of cold, foggy, and windy. But I’ve managed.
What’s surprising about growing tomatoes is how something so small can grow so big. I start with a tiny sapling in a thin plastic pot. I move it to a larger pot and then water it every couple of days. By the end of a week its mass has doubled. At a month it’s over a foot high and blossoming. After two months the first tomatoes are growing fast and the plant is chest-high.
Each day I water the plant it looks the same as the day before. I can’t notice daily changes because the differences are subtle. I went away for a week and when I got back the plant looked enormous compared to before. When I hadn’t seen the incremental changes, the plant’s growth was astonishing. Growing tomatoes has made me see the value solely in time passing.
How would I be if I spent a tiny bit of time cultivating myself every day? It wouldn’t seem like much to me, since I’d witness the small differences. It’d be hard to stick with it. But after two months or a year, I may look back and realize how far I’ve come. I think this is how I developed as a programmer. I wonder how else I could improve this way.
from One Big Fluke http://bit.ly/10q1q3E
Bad marketing email. Not even trying to be experimental about it. No call to action. Too much text.
from One Big Fluke http://bit.ly/199Urkn
