Google Civil Twilight
Dear users,


Google service which you


will be discontinued


Thanks for your support. Please try out



See you at http://ourincrediblejourney.tumblr.com/ !

Love, Google.


The Last Artisans
HERMAN      uh grew up watchin' mah pappy make modems with just
            a chisel an' hacksaw. course, things're different now...

BG MUSIC    drawn-out harmonica

NARRATOR    Herman McCowski is one of the last modem artisans in Tennessee.
            Modems are his life.

HERMAN      The hell is this!

Herman picks up an Ethernet cable that has fallen out of his
teenage son's backpack.

HERMAN      The hell is this, boy? Don't you walk away when I'm talkin'
            to ya!"

He waves at the camera angrily -- "Put that thing away. Put it away."

INT         Herman weeping quietly in his workshop


Communicating with a KRUSH Flip Clock dot matrix display

If you’ve bought a KRUSH Flip Clock dot matrix display, and you want to display your own arbitrary data on it, here’s the information you need:

Connect a serial cable through a rs485 adapter to the blue and white wires. Connect at 57600.

The clock is composed of two chunks: the top 28x7 and the bottom 28x7. Call these 0 and 1. Each column of 7 displays a byte in binary.

To write a chunk, send 0x80 0x83 (0 or 1, for chunk id) (all 28 bytes) 0x8F for a total of 32 bytes per chunk.

Thus the following fragment:

def send_picture(self, chunk, data):
    assert chunk in [0, 1]
    assert len(data) == 28
    l_bytes = [0x80, 0x83, chunk] + data + [0x8F]
    packed = struct.pack('32B', *l_bytes)

(Where sp is a serial connection to the device.)

Here’s a web server for it.

I hope this helps someone!



The longest night, the shortest day. When we gather around fires, and around evergreens, and tell stories about death and rebirth, to celebrate that it’s the darkest time again, and so it’s going to get lighter now.

And as well as to celebrate, we cuddle up to reassure ourselves and each other that it is, in fact, going to do that getting-lighter thing again. And maybe to have company while we worry, a little, that this time it might not.

But it always does.

A pendulum is fastest at the bottom of its swing, and pauses at the high points at the ends. But the year, I think, is slowest at the bottom of its swing, and down here at the Winter Solstice we feel everything pause in the darkness and the cold, everything holds its breath, motionless for a moment.

We hold our breaths, long enough to look around, make sure that the warm place, the bright place we’ve secured in the darkness, is all ready, all warm, all bright. So the moment of most-dark can pass easily, quickly, smoothly, with us quiet and watching, and after that breath-held dark-quiet pause, motion can start again, and later the sunrise, and not too long after that the Spring.

David Chess


Neurally Stylin

I’ve been playing with neural-style, a deep-learning algorithm that lets you apply the texture/style information from one image to the content of another image.

If you’d like to play with this on your own, and you’re comfortable with AWS, you can use the following script.

First, start an ami-ffba7b94 instance as g2.2xlarge. This is a Torch machine learning AMI. Currently, that costs $0.65/hour. If you do it as a spot instance, it’s way cheaper - more like $0.08 an hour. If you bid $0.10 or so, you’re very unlikely to get killed. This installs neural-style and creates a little shortcut doit script that takes input image, style, and size (pixels) as arguments. Don’t set size bigger than about 600 or it’ll run out of memory.

There are a ton more options you can set, read the neural-style docs.

# start a ami-ffba7b94 instance (Torch machine learning AMI)
# https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#LaunchInstanceWizard:ami=ami-ffba7b94
# on a g2.2xlarge instance. if you do it as a spot instance
# it is WAY cheaper and generally if you bid $0.10 or so you won't get killed.
# then run this script on your new instance and source .bashrc
# and you will have a nice little 'doit' script.
# make sure you source .bashrc or relogin after running this.

luarocks install image
luarocks install loadcaffe
luarocks install torch
echo "export LD_LIBRARY_PATH=/home/ubuntu/torch-distro/install/lib:/home/ubuntu/torch-distro/install/lib:/home/ubuntu/cudnn-6.5-linux-x64-v2-rc2" >> ~/.bashrc

git clone https://github.com/jcjohnson/neural-style
cd neural-style
sh models/download_models.sh

echo '#!/bin/bash

OUT=$(basename ${IMAGE%.jpg})_$(basename ${STYLE%.jpg}).png

th neural_style.lua -num_iterations 2000 -style_image $STYLE  -content_image $IMAGE -image_size $SIZE -backend cudnn -output_image $OUT
' > doit
chmod +x doit

The above link has some good examples; here’s some of my own output:

Ada-mew, applied to an abstract-ish image I got off etsy, The Starry Night, and a Louis Wain cat.

Celeste, applied to a Mandelbrot set, The Starry Night, and a Dr. Seuss image.

Celeste and Ariel, applied to Gustav Klimt.


at least it didn't say SQUEAK

I fed all of Discworld into a neural network, and got this back:

NOTE: This is pretty random. For the same training set, but with a lower temperature setting (which makes the model more confident but more conservative as well in its samples) see this output instead.

        ‘I’m going to stop a few sizes of Great Wizard on day, sir,’ said a voice behind Arthur, and this time he was worse for this about the sky. ‘It’s not, sir, yesterday I’m … a detail, on your face, wheezing off the barn. Low huge times. I was going to get for him, so this pig is a bit making me caring out as, and the common dancing slightly are a man. I don’t know.’

        ‘How do we see the gods?’ said Rincewind, who knew about that dog. He continued to Lancre. ‘I know the Funnied Summer One, Sir.’ Vimes sighed. ‘Ah, there were a lot of little stalls penny to me. Long stars thinking!’ Vimes cleared his throat.

        ‘Nanny, people are flying and even the girl who spent dwarfs, man,’ he added. ‘Pick through Captain Vimes’s instincts. The Dean has something vampires don’t seen Detritus in which was the other four. We want to personally know when he made the generals for Twoflower, who was only coming in to large.’

        ‘Miss Nitt,’ he said. ‘I never have been walking along the hall. Miss Nitt can get the idea about shapes, there’s some sort of lands and water an’ pointy father in that dark and a certainty of me… , yes, I am felt so sword and officer, you know, and I’d like to come back. The right its yet stupid common candle.’

        Is this far the ground?’ said the day Assassins. “Don’t take it. The last later and ram bore a large figure descendance around to my corpse. go-foot wrong and grows.

        The encouragement rose. Mr Groat did not even know the direction of the motion. Oh, yes. Some the noise of the Patrician’s nose didn’t believe in. ‘But a captain of the Horde were casting a start,’ said Moist, and then, all right, badly carefully moving. ‘While I want the first thing you know how to ask you to decide that we could get up on the station,’ he said. ‘That’s true! They will deducit you,’ said Moist. ‘Look, really. Sweet water. Bory horses, was there all lordship. Wonderful, reflective fingers, will you? I father, ye gods are the worse!’

– Zombie Pratchett


Ariel's Commute

How long does Ariel’s commute take, and does it matter when she leaves?

Ariel grabbed her data from Moves. Here I’m going to see how time-of-day affects her commute duration.

Side note: Kudos to Moves for excellent export facilities. Absolutely wonderful. They give you the data in every format and breakdown imaginable.

%pylab inline
import pandas as pd
import seaborn
df = pd.read_csv('storyline.csv')
df['Start'] = pd.to_datetime(df['Start'])
# we only want since the current commute started
df = df[df['Start'] > '2015-03-01']
Date Type Name Start End Duration
4897 3/1/15 place Home 2015-03-01 05:00:00 2015-03-01T11:17:27-05:00 40647
4898 3/1/15 move transport 2015-03-01 16:17:27 2015-03-01T11:49:20-05:00 1913
4899 3/1/15 move walking 2015-03-01 16:49:19 2015-03-01T11:56:58-05:00 459
4900 3/1/15 place Place in Cambridgeport 2015-03-01 16:56:59 2015-03-01T14:12:03-05:00 8104
4901 3/1/15 move walking 2015-03-01 19:12:03 2015-03-01T14:19:04-05:00 421

Later I’m going to only want the time of day, not the date, so set all dates to 1970-01-01.

df['Start'] = df['Start'].apply(lambda x: x.replace(year=1970, month=1, day=1))

I want to know which transport rows go between Home and Koch, so I make two new columns shifting name up and down so I can match against those.

df['prev_name'] = df['Name'].shift(1)
df['next_name'] = df['Name'].shift(-1)
Date Type Name Start End Duration prev_name next_name
4897 3/1/15 place Home 1970-01-01 05:00:00 2015-03-01T11:17:27-05:00 40647 NaN transport
4898 3/1/15 move transport 1970-01-01 16:17:27 2015-03-01T11:49:20-05:00 1913 Home walking
4899 3/1/15 move walking 1970-01-01 16:49:19 2015-03-01T11:56:58-05:00 459 transport Place in Cambridgeport
4900 3/1/15 place Place in Cambridgeport 1970-01-01 16:56:59 2015-03-01T14:12:03-05:00 8104 walking walking
4901 3/1/15 move walking 1970-01-01 19:12:03 2015-03-01T14:19:04-05:00 421 Place in Cambridgeport transport


koch = 'MIT Koch Day Care Ctr'
east = df.loc[(df.Name == 'transport') & (df.prev_name == 'Home') & (df.next_name == koch) ]
west = df.loc[(df.Name == 'transport') & (df.next_name == 'Home') & (df.prev_name == koch) ]
Date Type Name Start End Duration prev_name next_name
4945 3/5/15 move transport 1970-01-01 12:49:44 2015-03-05T08:45:36-05:00 3352 Home MIT Koch Day Care Ctr
4983 3/9/15 move transport 1970-01-01 11:50:58 2015-03-09T08:32:02-04:00 2464 Home MIT Koch Day Care Ctr
5059 3/16/15 move transport 1970-01-01 12:03:33 2015-03-16T08:40:40-04:00 2227 Home MIT Koch Day Care Ctr
5075 3/17/15 move transport 1970-01-01 12:16:17 2015-03-17T08:49:58-04:00 2021 Home MIT Koch Day Care Ctr
5093 3/18/15 move transport 1970-01-01 12:21:08 2015-03-18T08:51:00-04:00 1792 Home MIT Koch Day Care Ctr


    plot(west.Start,west.Duration,'.')  # not bothering to make the x axis a date for now


Let’s get rid of outliers.

Yes, obviously if you leave work in the middle of the day, there’s zero traffic, but we don’t want those 5 points driving the entire model, so let’s drop them.

west = west.ix[(west.Start > 7.5e13) & (west.Start < 8e13)].copy()




east = east.ix[(east.Start > 4.2e13) & (east.Start < 5e13)].copy()


How long does the commute take?

hist([k/60 for k in east.Duration], 15, [20, 60])
# using pyplot.hist instead of DataFrame.hist so I can /60 to get minutes


hist([k/60 for k in west.Duration], 15, [20, 60])


Write to file.


Now, do some statistics, in R

Thanks to @ozjimbob for most of this.

Load in the data and convert the times to a POSIXct class, then make them numeric because gam models don’t always play nice with this class. So we have seconds from the UNIX epoch.

I’m also going to convert Duration to minutes, because who thinks in seconds.

east=read.csv("east.csv", stringsAsFactors=FALSE)
west=read.csv("west.csv" , stringsAsFactors=FALSE)
east$Start = east$Start - 3600*4   # fix time zones back to US/Eastern
west$Start = west$Start - 3600*4
east$Duration = east$Duration / 60
west$Duration = west$Duration / 60

We’re going to do generalized additive models that allow you to fit a spline smooth through a predictor variable.

e_mod=mgcv::gam(Duration ~ s(Start),data=east)
w_mod=mgcv::gam(Duration ~ s(Start),data=west)

Let’s examine the summaries for these.


## Family: gaussian 
## Link function: identity 
## Formula:
## Duration ~ s(Start)
## Parametric coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  35.4346     0.8483   41.77   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## Approximate significance of smooth terms:
##            edf Ref.df     F  p-value    
## s(Start) 6.574  7.712 6.318 1.18e-05 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## R-sq.(adj) =   0.46   Deviance explained = 52.7%
## GCV = 45.195  Scale est. = 38.856    n = 54


## Family: gaussian 
## Link function: identity 
## Formula:
## Duration ~ s(Start)
## Parametric coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  41.2279     0.8465    48.7   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## Approximate significance of smooth terms:
##          edf Ref.df     F p-value
## s(Start)   1      1 0.139   0.711
## R-sq.(adj) =  -0.0195   Deviance explained = 0.316%
## GCV = 34.463  Scale est. = 32.965    n = 46

West looks terrible - there’s just nothing there. Let’s start with East.

East - morning commute to work

Let’s look at the east model - generate predictions, with standard errors, for each possible time between the minimum and maximum.


e_predict=predict(e_mod,newdata = pred_frame,se.fit=TRUE)

pred_frame$upper=e_predict$fit + e_predict$se.fit
pred_frame$lower=e_predict$fit - e_predict$se.fit

Now prepare a graph - plot the predicted fit with the error bars. Convert the times back into POSIXct for nice plotting too.

pgoe_x = c(pred_frame$Start,rev(pred_frame$Start))
pgoe_y = c(pred_frame$upper,rev(pred_frame$lower))


plot(pred_frame$pred ~ pred_frame$Start_po,type="l",ylim=c(20,60),xlab="Start",ylab="Duration")
points(east$Duration ~ east$Start)


Let’s try a regression tree.

rp_mod=rpart(Duration ~ Start,data=east)
pred_frame$pred_rt = rp_predict
plot(pred_frame$pred_rt ~ pred_frame$Start_po,col="red")
points(east$Duration ~ east$Start)


And a randomForest.

rf_mod=randomForest(Duration ~ Start,data=east)
## Call:
##  randomForest(formula = Duration ~ Start, data = east) 
##                Type of random forest: regression
##                      Number of trees: 500
## No. of variables tried at each split: 1
##           Mean of squared residuals: 53.52246
##                     % Var explained: 24.24

Plot the model predictions.

pred_frame$pred_rf = rf_predict
plot(pred_frame$pred_rf ~ pred_frame$Start_po,col="red")
points(east$Duration ~ east$Start)


It’s insanely overfitted, but ok.

West - evening ride home

Let’s look at the west plot.


Ariel leaves work in such a narrow window that there’s just no difference in the small changes she makes. Maybe in the future she can start experimenting with intentionally leaving later to see what that does to her commute time.

Everything’s going to be garbage, but let’s look anyway. Regression tree: center


## Call:
##  randomForest(formula = Duration ~ Start, data = west) 
##                Type of random forest: regression
##                      Number of trees: 500
## No. of variables tried at each split: 1
##           Mean of squared residuals: 49.80263
##                     % Var explained: -57.45

lol negative variance


lol nope


A hole in my eye

Today during a routine eye exam I noticed that I had a blind spot right in the center of my left eye - when I looked directly at a letter on the chart, it would vanish.

Now, I know that the fovea is worse at some kinds of vision; for instance, if you’re trying to look at a dim star, it’s best to look away and use your peripheral vision. But under normal circumstances, you should be able to see black text on a white background just fine in your fovea, yet that was disappearing for me.

So, they did an OCT scan of my eye, and lo and behold, I have a lamellar macular hole.

Good right eye:

Not so good left eye:

I’m told “it probably won’t get any worse, but it won’t get any better, either” and it’ll be rechecked every year.


High Fi

A story my dad used to tell:

My friend Ken worked for what was then a first-tier Japanese hifi company. At the time – it was the eighties – companies exhibiting at CES gave shit away. Shirts, pens, calculators… you know: trade show crap. Rather than have his giveaway get lost in that sea of crap, my friend decided to give out pocket-sized mirrors, each encased in a blue silicon sleeve emblazoned with the company logo. His Japanese masters said “Ken-san, why you give away small mirror. We don’t understand.” And Ken-san said “trust me, they’ll love it.” And they did, because a) it was the eighties, and 2) while some preparatory activities are best done on a cardboard record jacket, others require an unyielding surface. I should add that the notion of including a single-edge razor blade in the package crossed his mind, but was rejected as being too on-the-nose.



Ariel is reading Hofstadter’s Le Ton Beau de Marot and wrote her own version of the poem:

Liza dear,
peer to peer,
hello world!
Here unfurled:
Jacquard’s loom.
Chinese room
you’re inside.
Do not hide -
Please recupe!
Close your loop
and proceed.
Like form-feed , without jams.
Don’t trust scams,
swear you’re real.
That appeal
Said three times
Ends my rhymes.
Clement here,
Liza dear.


Facebook photo tags export

A few months ago, I scanned a few thousand old family photos and uploaded them to Facebook. Family members and friends came out of the woodwork from all over the world and helped tag them, identifying faces I barely remembered from childhood or had never known at all.

I wanted to export those photos back out of Facebook, with all those tags intact. Fortunately, https://developers.facebook.com/docs/graph-api[Facebook's API] is incredibly rich and easy to use - Facebook may be a walled garden, but it’s one that’s really, really easy to copy your data out of. (Note I didn’t say remove.)

I used the facebook-sdk.

import facebook
token = 'CAACEdEoseetcetcmysecrettokenwasherezEwZDZD'
g = facebook.GraphAPI(access_token=token)

First I went to Facebook by hand and got the album ids I wanted to download from.

albumswanted = ['10101527783396627',
                '10101527672838187'] # in real life there were a lot more

Next, I set up a dict where I will store the data. For each of the albums, I want to get all the photos in it. Facebook is super-nice and actually returns all the data I need in this query (I can imagine an alternative world in which it just returned a bare list of photoids which I would have had to execute subqueries on).

If the current page returned by get_object has data, I print out the photo’s id, its source which is a URL for the image, and then if the photo has any face tags, I print them.

albums = {}
for album in albumswanted:
    curpage = g.get_object('{}/photos'.format(album),

If the page has data, it will have a length of 2; a data key and a paging key. When finished, curpage will just have an (empty) data key, and so it will be of length 1.

    while len(curpage) == 2:
        for photo in curpage['data']:
            if 'tags' in photo:
                print('{}\t{}\t'.format(photo['id'], photo['source'])),
                for tag in photo['tags']['data']:

When I’ve run out of photos, grab the next page of data, if one exists, and do it again.

        curpage = g.get_object('{}/photos'.format(album),

This script outputs lines like:

    10101527734988637	https://scontent.xx.fbcdn.net/hphotos-xpf1/t31.0-8/s720x720/10511474_10101527734988637_3611385766771831953_o.jpg	"Dede Shanok Drucker"	"Daniel M Drucker"

I saved the output of that script to tags.txt. Next, I grabbed all of those image files:

awk '{print $2'} tags.txt | wget -i -

Note, however, that the files I retrieved are not the full resolution images! Yes, I could have retrieved the full resolution images, but that wouldn’t actually help me - Facebook strips original Exif information, and in any case, I want to identify the subset of these images I have on my local disk that have been tagged on Facebook.

I now have a folder facebook full of images I just downloaded. I also have a folder originals which contains (a larger set of) images, with different filenames and higher resolution, some of which correspond to the images in the facebook folder. I want to determine a mapping between the two.

Because the image transform is so simple (just resizing, no other transformation), really any image comparison tool would probably work - even something incredibly naive like histogram matching. I ended up using the first thing I came across, the pHash library, and it worked great:

import phash
import glob
import itertools
import sys

thumbhash = {}
orighash = {}

for fn in glob.glob('/path/to/facebook/*.jpg'):
    print('hashing {}'.format(fn))
    thumbhash[fn] = phash.image_digest(fn)

for fn in glob.glob('/path/to/originals/*.jpg'):
    print('hashing {}'.format(fn))
    orighash[fn] = phash.image_digest(fn)

I use itertools.product to check every hash in the facebook images against every hash in the originals. It’s a cartesian product, so it’s a large number of comparisons, but it’s actually quite fast - the slow part was computing the hashes in the first place.

for element in itertools.product(thumbhash,orighash):
    cc = phash.cross_correlation(thumbhash[element[0]],orighash[element[1]])
    if cc > .99:        # it should really be pretty perfect

This generates lines like

('10628778_10101527712643417_5250594863317404614_o.jpg', '/home/dmd/Dropbox/scan2014/feb93/Feb93alb_p_026.JPG')

Using those and the tags from tags.txt, I can then just use Exiftool to add the tags I want to the right image files!


Real Estate Disclosures

I’m starting to keep track of what the MLS Disclosures for a house should have REALLY said.

DISCLOSURES: Architect basically phoned it in.

DISCLOSURES: If you have a shred of soul in you it will be eradicated by taking one step into this home.

DISCLOSURES: There is probably a reason this property has been bought and sold 3 times in 10 years.

DISCLOSURES: You only WISH that were black mold.

DISCLOSURES: Previous owner used kitchen primarily for cooking meth.


Tazo Sweet Cinnamon Spice

Ariel says the tea I’m drinking smells like “ebay clothes from a seller with a religious name”.


Generation gap: How communication modes are perceived by different ages.
up to 30 30 – 50 50 and up
phone call What’s a phone call? Rude. Interrupting, presumptuous. Polite.
email Rude. Long-winded, waste of time. Polite. Rude. Impersonal.
text/tweet Polite. Rude. Brusque, impersonal. What’s a text?


calendrical calculations

I’m trying to define a new [silly] calendar, because there aren’t enough of them yet.

My calendar, so far, is specified:

epoch (orthodox) is the moment the leading edges of the dinosaur-killing asteroid touched the surface of the earth.

As this is somewhat difficult to define, for practical purposes we’ll use:

epoch: same as the unix epoch, i.e., 1970-01-01T00:00:00Z.

From that point, start counting seconds in successive prime-numbered sequence; these are termed ‘dinosecs’, such that epoch + 2s = dinosec 1, epoch + 2s + 3s = dinosec 2, epoch + 2s + 3s + 5s = dinosec 3, dinosec 3 + 7s = dinosec 4, etc etc.

It is, however, forbidden to use seconds in the actual specification of time - all time must be referred to in dinosecs and fractional dinosecs.

Here’s my question: What is a good way to go about defining fractional dinosecs?

One obvious way is to simply take the number of seconds since the last dinosec and divide it by the number of seconds between that dinosec and the next one. I dislike that method because it makes dinosecs nondifferentiable at whole-numbers - i.e. there’s a sudden change in how quickly the fractional part is incrementing.

Is there something that could be done with x/ln(x), for example? Or is there an more rigorous way to do it?

Any ideas would be appreciated.


It's a subset of everything.

I had a dream just before I woke up where I was explaining to someone that “this store has everything you can imagine - as long as your imagination mostly expresses itself in Hummel figurines.”


Where I've been, redux.

I made a heatmap of all my USA Flickr photos using Python.


Just in case.


Just in case.



epitome. noun. a large dusty heavy book carried on the person for the purpose of squashing bees before they sting you. now largely superseded by the epipen.


China Miéville's ZORK

Every time. I go to bed and every time it happens. There I am, standing in an open field. There's a white house with a boarded up door. And every time there's a mailbox.

Sometimes I think I was the one who boarded up the door. Other times I wake up and I can see the boards closing over me.

– by Ariel