Login

Tags

Factual Blog / Tagged:

spencer tipping

Investigating Various Pathologies of Low Quality Location Data #1 - App Permissions - Lab Notes

Note: This is a companion post to Investigating Various Pathologies of Low Quality Location Data #1 - App Permissions Repeated latitude digits and where they come from Audience data validation is a crucial part of delivering accurate behavioral profiles, and as such we put a lot of effort into understanding the sources of inaccuracy in location...

Bug Du Jour: CDH5 Upgrade

We upgraded our Hadoop cluster to YARN/CDH5 last weekend, which brought along the usual flurry of “oops, gotta fix this” commits as various services had hiccups, and in many cases refused altogether to do anything useful. Last week Tom sent me my favorite message: “I just want this to work” (seriously, it’s awesome to get these because you...

A Day in the Life of a Factual Engineer: Polygon Compression

In this series of blog posts, Factual’s blog team asked engineers to describe what a typical day looks like. Background Chris Bleakley, our resident polygon and Lucene expert, had written meticulous documentation about the problem he was solving. The first paragraph read: “Because search times are dominated by the cost of deserializing JTS objects from when...

How Geohashes Work

We use geohashes all the time at Factual, so one day I got curious and read through the canonical Java implementation to figure out exactly how they work. It was an enlightening read, but along the way I encountered some unfortunate bits of coding horror that got me wondering about the fastest way to encode and decode geohashes. The geohash...

The Humongous nfu Survival Guide

Github: github.com/spencertipping/nfu A lot of projects I’ve worked on lately have involved an initial big Hadoop job that produces a few gigabytes of data, followed by some exploratory analysis to look for patterns. In the past I would have sampled the data before loading it into a Ruby or Clojure REPL, but increasingly I’ve started to...