FatTuba.com

Improve Email Deliverability From Debian

2024-11-03T00:00:00-05:00

When you say that you want to host your own email, you’ll hear the standard line from folks: “Don’t do it. It’s too hard.” When they say “it’s too hard” what they mean is that it’s too hard to ensure that email you send will actually end up in other people’s inboxes.

If you persist, they’ll advise you to configure your email server with SPF, DKIM, and DMARC. Oh and also make sure the IP address of your email server isn’t on a blacklist.

After you complete these recommended incantations and start using your shiny new email server, you’ll notice that your email recipients on gmail don’t see your emails in their inbox. After reaching out, your recipients will tell you your email ended up in their gmail spam folder. So you start searching the internet for answers. “How do I keep my email from ending up in gmail spam folder?” The answers all pretty much point back to the incantations above. So to double check your setup you use a service like mxtoolbox and everything looks pretty good.

So now what? Do you give up on the distributed nature of SMTP and let Google/Apple/Microsoft oligopoly host your email like 90% of the internet? Be stubborn and resign yourself to the fact that you’ll need to utter the words “have you checked your spam folder” on a regular basis?

<sarcasm>I’m sure it’s shocking to folks that know me</sarcasm> that I’m squarely in the second camp: stubborn and stuck in spam folders.

Because I’m a counter-cultural rebel, I’m burying the click bait tag line here instead of in the title: Here’s one weird trick to improve email deliverability!

Hide your Received headers. That’s it. Once I started doing that a few days ago, my email started showing up in my buddy’s inbox instead of his spam folder. It makes sense I suppose. Google looks for suspicious IP addresses sending email so even though you’re relaying email through a smarthost with a trusted IP address, your untrusted dynamic IP from your home ISP is still visible in a Received header. So hiding those headers leaks less information to Google that they can use against you.

Here’s how to hide Received headers if you’re using Exim on Debian as your email server:

me@myemailserver:~$ grep Received /etc/exim4/conf.d/main/000_localmacros
REMOTE_SMTP_TRANSPORTS_HEADERS_REMOVE = Received

Anyway, enjoy folks actually receiving your email!

Pry URLs From My Cold, Dead Hands

2020-11-16T00:00:00-06:00

Google is trying to hide the URL in Chrome. This has been tried before in various browsers at various times.

I’m deeply unhappy about this, not because of Google’s move in Chrome but because of the general trend in the industry to make the URL a second class citizen.

The current worst examples of this is an Electron app that I use everyday: Slack. When I run Slack in the browser I get the benefit of being able to see multiple conversations at a time(multiple browser windows), or be able to jump between conversations quickly(multiple browser tabs). Another advantage of running Slack in the browser is that I can easily share links to comments/channels without having to dig around for the “Copy Link” or “Share” button.

In fact, the only reason to fire up Slack as an Electron app is to share my screen. Loading Slack in the browser wins every other time ‘cause URLs.

Markdown for Microsoft Word

2019-08-17T00:00:00-05:00

If you prefer text-based documentation(markdown, etc) over WYSIWYG(MS Word), then this is for you! If you work in modern corporate America, Microsoft Word is unavoidable. Even though I wish it weren’t so, most engineering organizations use it for technical documentation.

I want to use markdown as the source and then automatically render Microsoft Word. Most folks online recommend pandoc for this kind of thing. The problem is that pandoc renders docx using it’s own style.

But there’s usually a standard corporate template that you’re required to use.

At this point I usually stare blankly at my screen for a couple of minutes then begrudgingly load up the corporate template in Word and start copy-n-past’ing then manually formatting. But…. no…. longer.

I came across a little hack that works pretty well. If you insert unstyled HTML into a Word template using Insert->Object->Text from File, Word will automatically apply the template styles.

So, I generate HTML from markdown: pandoc -s my.md -o my.html, then insert the HTML using the Text from File menu item in Word and voila! I have a perfectly formatted Word docx using the corporate template automatically.

However, it’s still a hassle to have to perform these manual menu selections everything I want to re-render the docx. So here’s a little Python script to automate that part using COM:

import sys
import os
import argparse

import win32com.client as win32
from win32com.client import constants as c


def main(argv=None):
    if argv is None:
        argv = sys.argv

    parser = argparse.ArgumentParser(description="convert html to docx")
    parser.add_argument("htmlfile", help="the html file path for input")
    parser.add_argument("docxfile", help="the docx file path for output")
    parser.add_argument("-t", "--template", default="template.dotx", help="the template file")
    args = parser.parse_args(argv[1:])

    template_path = os.path.realpath(args.template)
    html_path = os.path.realpath(args.htmlfile)
    docx_path = os.path.realpath(args.docxfile)

    word = win32.gencache.EnsureDispatch('Word.Application')
    try:
        doc = word.Documents.Open(template_path)
        word.Visible = True
        section = doc.Sections(2)
        section.Range.InsertFile(html_path)
        doc.TablesOfContents(1).Update()
        for i in range(1, doc.InlineShapes.Count + 1):
            doc.InlineShapes(i).LinkFormat.BreakLink()
        doc.SaveAs2(docx_path, FileFormat=c.wdFormatXMLDocument)
    finally:
        word.Quit()


if __name__ == '__main__':
    sys.exit(main())

It also regenerates the table of contents and embeds the images after the text is inserted. I hope this helps any other text-based documentation lovers out there.

2019/9/13 Update: We’re now embedding all the images instead of leaving them as links.

2020/6/15 Update: It turns out that this method doesn’t work in the general case. It happens to work with my template because the template fonts are close enough to the default Word fonts that they match.

2024/9/14 Update: Perhaps this is the best answer now?

A Solution for DDoS

2016-10-09T00:00:00-05:00

If you haven’t heard the news yet, there was a record DDoS attack perpetrated against Brian Krebs last month. Distributed denial of service attacks are nothing new. But the particulars of attack changing. In the past, DDoS attacks used reflection or amplification techniques to take a large botnet and compound their effective attack traffic. The big deal with this new attack is that the botnet was much bigger than previous botnets. So big, in fact that reflection and amplification hacks weren’t necessary. The purported size is 1 million devices. And the botnet was comprised mainly of IoT devices such as security cameras.

“Botnet” by tecnomovida is licensed under CC BY-NC-SA

That has all been well-covered by the tech press. The question is, how do we prevent or at least reduce these kinds of attacks? As usual, follow the money!

The real issue is one of economics and not technology. These DDoS attacks come from p0wned devices on users’ networks. Right now the service providers(e.g. Brian Krebs) pay per byte for their bandwidth but users do not. Most home and business internet connections are not billed per byte. So most users don’t care how much traffic they generate.

Consequently since the users aren’t paying for the traffic, there’s no economic incentive for them to keep their devices up-to-date with security patches.

What if users’ internet was billed per byte? If a user’s device were a part of a DDoS botnet, their monthly internet bill would increase due to the increase in traffic coming from the hacked device. The user would then be more likely to try to find the rogue device and take it offline or clean it. Subsequently, users would start demanding that manufacturers create more secure devices that are patched on a regular basis.

Update: 2026-1-25

Because the economic incentive is widely distributed among the device owners relative to the benefit to the DDoS attacker, the plan above probably won’t work. It’s the computer security equivalent to a salami attack.

IoT Platform Dominance

2016-05-13T00:00:00-05:00

There’s a lot of talk about the IoT platform, what that is exactly and what companies will end up winning. Well, I think it snuck up on us without us realizing it. There are really two levels of IoT devices.

Super low-powered - These are typically powered by a coin cell battery and run code on bare-metal.
Higher powered - These are typically plugged in or Li-Ion powered and run Linux.

The development kits and ecosystems around each of these device types are the platforms.

What Platforms Are Winning?

Both of these device types already have dominant players, at least in-terms of prototyping. Arduino is the leader amongst the low-powered devices while Raspberry Pi leads in the higher powered area.

“A hand-soldered Arduino” by Matt Biddulph is licensed under CC BY-SA 2.0

Why Are These Winning?

I think the reasons that these platforms lead are pretty simple.

First, I can buy them on Amazon. Easy access to these devices make them a great choice for prototyping. You don’t have to call a sales rep or navigate some vendor’s horrible estore to get them.

Second, most of the code I need is already available. The communities for both the Pi and Arduino are massive. There are tons of howtos and tutorials with code included that do everything from talk using Bluetooth from a Phone to adding a touch screen. The mind share around this stuff is really insane.

Finally, the hardware is open-ish. One of the things to consider when choosing a prototyping platform is how easily can you move to production. Arduino is legitimately open hardware, so I can take a duct-taped prototype to a PCB shop and they can quickly and easily give me a custom PCB based on the prototype. Most PCB shops have done Arduino-based boards before so it’s like a walk in the park. Although the Raspberry Pi isn’t open hardware they are moving in that direction. Also, there are shops that specialize in taking Pi prototypes to production.

“Raspberry Pi” by cowjuice is licensed under CC BY-SA 3.0

What Companies Win Here?

Broadcom makes the SoC on the Raspberry Pi and Atmel makes the micro controller on the Arduino so I think those guys have an advantage in the market moving forward. Of course all this depends on my assumption that easy prototyping translates into massive chip sales in the long run. What do you think?

IoT Usefulness

2016-04-06T00:00:00-05:00

Ok so why haven’t all my geek friends put smart lighting everywhere in their houses? I think it comes down to usefulness or bang for the buck.

For the purposes of this discussion, let’s define IoT as connecting a physical thing that hasn’t traditionally been connected in the past.

Ok, so after talking with Harlan about IoT in general(he’s speaking next Wednesday on it), we came up with the following four categories of IoT usefulness:

saves me time
saves me money
entertains me
makes me healthier

“The Internet of Things” by wilgengebroed is licensed under CC BY 2.0

An IoT product must provide benefit in one or more of the four categories above otherwise, it won’t sell. Here are some examples:

Nest thermostat - saves me money
Smart prompt kitchen thermometer - saves me time
Tricella pill box- makes me healthier

I am specifically excluding media devices from the IoT space(TV, radio, speakers) since those have been traditionally connected for a while now.

Aside: Amazon Echo isn’t an IoT device, but instead a new interface(voice instead of screen) to any services(IoT or non-IoT) that a screen would traditionally control. Does combining voice interface with some IoT product suddenly make that IoT product useful in one of the four categories where it wasn’t before?

Regarding the unified dashboard idea, how does pulling together the disparate sources of IoT data make those IoT products hit the threshold in one of the categories?

Anyway, I think a framework like this is useful for discussion. Is the framework presented the right way of thinking about IoT?

Android is GMS

2015-11-07T00:00:00-06:00

Well friends. I think it’s time to call it like it is. Android is Google Mobile Services. You see, Google would like us to think that Android is an open source mobile operating system that anybody can use and extend for free to do whatever they want. Here’s the hitch: the Google Play Store isn’t part of Android open source. The Play Store(along with other Google apps(YouTube, Maps, GMail, etc)is part of what Google calls Google Mobile Services or GMS.

And GMS is not open source. Use of GMS requires a separate license from Google. And while that license doesn’t cost anything, it’s almost impossible to obtain for most developers.

You see, we’ve built an Android device this year just in time for Christmas. We really worked hard to put together the best recipe reader we’ve ever done(fastest, crispest screen, etc).

The only problem: no Google Mobile Services. We couldn’t even get Google to talk to us about GMS licensing. Here was the response that we got:

Hello Ryan,

Thanks for your interest.

We are very swamped for direct license and ask partners work with our authorized ODMs. Attached the ODM list. If you can ask your manufacturing to either of them, they can bundle Google applications per our approval.

Regards

Ok, we thought, we’ll just build our new reader using one of the parter ODMs. The problem is that the hardware available from those partners is just average! We are committed to shipping the fastest clearest reader possible and we weren’t willing to sacrifice speed or screen clarity.

So what does that mean? That means we have a killer reader that uses the Amazon Underground App Store instead of Google Play. Now the Amazon Store does have some advantages over the Play Store: free apps that otherwise cost money in Google Play, easy access to Amazon video, etc. But is that enough? How will the market react? I think once the reviews start coming in we’ll know.

What do you think? If you can’t download apps from the Play Store, is it still Android? I’d love to discuss.

The Future of Mobile Ads

2015-10-07T00:00:00-05:00

Mobile advertising is at a crossroads. Apple has just enabled ad blockers in iOS. And users are realizing how much mobile data is being consumed by ads. Since Apple has little vested interest in web ads and a huge interest in building the best product so that users will shell out a premium for them, ad blocking makes sense in iOS.

Mobile ads can make pages load up to 3 times slower and consume 6 times the bandwidth of the same page without ads. Since the ads don’t have any immediate value for the user, the web is a better experience without the ads. So, from a user’s perspective, “no ads” is better.

Now for the other perspective. I work for a company that makes part of their revenue through mobile advertising. Like most sites in the U.S., a huge percentage of our mobile traffic comes from iOS. Enough has been said about ad blockers causing the potential downfall of the free web. To summarize, the long-term value of ads for the user is that the content can stay free. But instead of rehashing all that, let’s discuss possible solutions. To quickly re-summarize the problem, mobile ads:

cost users real dollars in the form of mobile data
don’t provide any near-term value for users
slow down the web and make for a worse experience

The ultimate solution fixes all three of these. So, let’s serve ads that:

don’t consume much mobile data
provide near-term value for users
load quickly so that the web is still snappy

The obvious solution is to do advertorial content instead of the usual display ads. This is content that is potentially valuable in the near-term for users because it provides useful information while at the same time pitching a product or service. It also doesn’t consume any more mobile data than a normal non-advertorial piece of content. And since there don’t have to be 3rd-party ad networks involved, no extra code needs to be loaded so the site stays snappy. The downside to advertorials is that it takes real work to create them. Someone has to write the content.

Got other ideas? I’d love to discuss.

Agile Hardware

2015-09-02T00:00:00-05:00

At Key Ingredient, we’re pretty heavily invested in Agile, Scrum in particular. This year we really wanted to try to ship some hardware in an Agile fashion. So after hiring a team and discussing, we decided that Kanban was right for the hardware projects. The first hardware project we tried was our little smart pan. The idea is that we’d ship a new prototype every two weeks to dogfood.

By using off-the-shelf parts, we were able to do the first prototype in 14 days.

Our Mark I prototype(I’m a Marvel Fan), was built using Raspberry Pi B plus some other stuff from Adafruit. It just did simple temperature measurement and alerts. Mark II was just a cleaned up version of Mark I. We did a better job of hiding the wires.

Over time, we added additional features including boiling and burning detection.

Mark V even included a built-in touchscreen.

Overall, the project has been super fun and most importantly has allowed us to learn a ton about what it takes to make a connected kitchen product.

Typing

2015-04-27T00:00:00-05:00

I’m shocked that I need to actually say this. If you work with computers(sysadmin, programming, whatever), learn to touch type! I had a 15 minute conversation with a college student about whether he should go into network administration or security auditing. After we finished talking, he turned around to his keyboard and started two finger typing! I’m talking hunting and pecking folks.

That’s like a carpenter’s apprentice repeatedly hitting himself in the thumb with his hammer.

If you buy the whole thing about software engineers being craftsmen, then your keyboard is a tool like your text editor, compiler, debugger, etc. You need to know how to use it. And there’s really no excuse. There’s about a million free typing tutor sites. Pick one and start.

There’s even a version of House of The Dead where you kill zombies by typing Japanese words floating above their heads instead of shooting them.

So you really have no excuse :)

Temptation Bundling

2015-03-18T00:00:00-05:00

Freakonomics Radio has a great episode on temptation bundling. Temptation bundling is the idea that if you have a chore that you hate to do, you should “bundle” it with something that you like. The classic example is watching your favorite TV show only when running on the treadmill. The advantage of temptation bundling over some other similar techniques is that it makes both activites better. You don’t feel guilty that you’re wasting time watching TV and you actual enjoy running on the treadmill!

Tasker Is the Killer Android App

2015-02-18T00:00:00-06:00

As far as I’m concerned, the killer app for Android, is Tasker. I don’t think there’s anything like it for iOS. Tasker allows you to associate various events with actions(sort of like IFTTT). So if I’m at work, tasker sets my phone to vibrate. “At Work” is defined as: Wifi is associated with the KeyArt access point(our AP at work). Here are some example of events:

wifi related - a certain access point
bluetooth related - connected to a certain device
time related - within a time range
GPS related - within a radius of certain coordinates

…and example actions:

start an app
stop an app
adjust volume

So here’s a more complicated and incredibly useful example. When I’m leaving work, I automatically start Waze and start navigating home. “Leaving Work” is defined like this:

If I’m connected to my car bluetooth and…
I’m within 500 ft of my work and…
It’s between 4 pm and 9 pm in the evening

Starting Waze and automatically navigating home is just an Android intent and Tasker can send those as actions.

Now that I’ve automated much of the routine interaction with my phone it would be hard to go back.

My Dev Setup

2014-12-08T00:00:00-06:00

My work setup right now is a Thinkpad W530 running Debian Wheezy. I’ve been pretty happy with it. I’m using chroots with Debian Unstable and the latest Ubuntu to install any newer userspace programs I need. I’m using VirtualBox to run Windows 7 and Microsoft Office(under duress).

The machine is pretty reliable. My current uptime is 47 days.

tubaman@cuisine:~$ uptime
13:35:32 up 47 days, 2:44, 9 users, load average: 0.57, 0.72, 0.63
tubaman@cuisine:~$

I have had a few issues though.

Hardware Issues

Overall the W530 is a good machine. However, the ribbon cable that connects the web cam keeps coming loose. That means that the web cam just stops working. I guess the repeated opening and closing of the laptop causes the cable to work free. To fix it I had our resident hardware hacker crack it open and hot glue the cable into place. Disappointing but nonetheless fixable.

The other issue I have with the hardware is that the plastic case has developed a crack in the corner. I had a similar problem with my old Thinkpad T40p. This is where I get a little jealous of my Mac friends with their all-aluminum case.

Software Issues

The W530 comes with an on-board Intel graphics chip plus a discrete NVidia graphics card. The NVidia guys call this Optimus. It allows you to run on the lower-power on-board graphics until you’re playing a game or something. That’s when it switches on the discrete graphics. The drivers for Linux are not quite as seamless as the Windows drivers yet. However the Bumblebee Project has a tool called bbswitch that works pretty well. I use that to enable the external monitor. Oh yeah, did I mention that the external monitor port is connected to the discrete graphics chip?

VARK and UNIX

2014-11-27T00:00:00-06:00

On the recommendation of our product owner, everyone on the team is taking the VARK quiz. It’s designed to tell you what your learning style is. The options are visual, aural, reading, and kinesthetic. Here are my scores:

Visual - 0
Aural - 5
Read/Write - 4
Kinesthetic - 7

I wonder if those of us who gravitate towards the command line instead of pretty UIs have similar scores?

Android Record and Playback

2014-10-24T00:00:00-05:00

There are a bunch of suggestions online about how to record a series of touchscreen touches and play them back. None of them really work for me though. I’m looking for something that doesn’t require another app and optimally would work over USB debugging with adb. So I’ve hacked something together myself. I started out by just capturing all the events from my touchscreen like this:

adb shell getevent /dev/input/event1 > events.log

The events look like this:

0003 0039 0000009f
0003 0039 00000000
0003 0035 00000142
0003 0036 00000189
0000 0000 00000000
0003 0039 ffffffff
0000 0000 00000000

Those columns are the event type, code and data, all in hex. I used a little python script to reformat them into sendevent calls that look like this:

adb shell sendevent /dev/input/event1 3 57 159
adb shell sendevent /dev/input/event1 3 57 0
adb shell sendevent /dev/input/event1 3 53 322
adb shell sendevent /dev/input/event1 3 54 393
adb shell sendevent /dev/input/event1 0 0 0
adb shell sendevent /dev/input/event1 3 57 4294967295
adb shell sendevent /dev/input/event1 0 0 0

That works pretty well but doesn’t take into account the delay between each event so I modified my event capture to prepend a timestamp:

adb shell getevent /dev/input/event1 | ts "%Y-%m-%d %H:%M:%.S" > event.log

Then I modified my python script to add sleeps in between the sendevent calls:

adb shell sendevent /dev/input/event1 3 57 159
adb shell sendevent /dev/input/event1 3 57 0
adb shell sendevent /dev/input/event1 3 53 322
adb shell sendevent /dev/input/event1 3 54 393
adb shell sendevent /dev/input/event1 0 0 0
sleep 0.041
adb shell sendevent /dev/input/event1 3 57 4294967295
adb shell sendevent /dev/input/event1 0 0 0

Now when I execute that I get an exact playback(or pretty close)! Here’s the final record script:

1
2
3

#!/bin/bash

adb shell getevent /dev/input/event1 | ts "%Y-%m-%d %H:%M:%.S"

…and the python script that converts the output of the record script to something that can get played back:

#!/usr/bin/env python
import sys
import re
import datetime

line_re = re.compile(r"^(\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d\.\d+) (\S+) (\S+) (\S+)$")
event_log = sys.stdin
last_ts = None
for line in event_log:
    match = line_re.match(line.strip())
    ts, etype, ecode, data = match.groups()
    ts = datetime.datetime.strptime(ts, "%Y-%m-%d %H:%M:%S.%f")
    etype, ecode, data = int(etype, 16), int(ecode, 16), int(data, 16)
    if last_ts:
        delta = ts - last_ts
        sleeptime = '%.3f' % (delta.total_seconds())
        if float(sleeptime):
            print "sleep %s" % sleeptime
    last_ts = ts
    print "adb shell sendevent /dev/input/event1 %s %s %s" % (etype, ecode, data)

So now, to record I do:

./record > event.log

…and to playback I do

./convert_event_log_to_sendevents < event.log | bash

Bit by the POODLE

2014-10-22T00:00:00-05:00

Well, the internet was all abuzz about POODLE and it finally bit me. I use selenium with phantomjs to do some web scraping. One of the sites I scrap uses SSL and they removed SSLv3 because of the POODLE vulnerability. When loading a page with phantomjs, it would not load but instead stay on the “about:blank” page. All the Googles say that I should just set --ignore-ssl-errors but that didn’t help. I realized that it was probably an SSL protocol issue when I saw that phantomjs uses only sslv3 by default. You have to tell phantomjs to use other versions like this: --ssl-protocol=any.

State of the Art Web Scraping

2014-09-27T00:00:00-05:00

My go-to toolkit for web scraping is python requests and BeautifulSoup4. In combination, they provide a nice, clean API to interact with web sites and parse content. However recently I’ve had a tough time logging into some of the more complicated web sites(ahem citibank). The problem is that citibank uses javascript to calculate tokens on the client that are then passed to the server for checking. If the tokens don’t match, the login fails. I have to replicate these calculations in python for my scraper to be able to login. What a pain!

There are tools that are great for this kind of thing. I like selenium using the phantomjs webdriver. The problem is that while selenium is great for complicated logins, it’s terrible at things like downloading files or XHR JSON data.

I need a way to glue requests together with selenium so I can use the right tool for the right job: selenium for login and requests for everything else. Cookie copying to the rescue! I use selenium to login, then copy the cookies to requests for everything else. Here’s the snippet to copy the cookies:

def copy_cookies_to_session(driver, session):
    """Copy cookies from selenium webdriver to requests.Session"""
    cookies = driver.get_cookies()
    for cookie in cookies:
        session.cookies.set(
            cookie['name'],
            cookie['value'],
            domain=cookie['domain'],
            path=cookie['path']
        )

Now I can use the right tool for the right job.

Debugging Encrypted Web

2014-09-26T00:00:00-05:00

Debugging encrypted web traffic can be challenging. You have Chrome Developer Tools but I find that being able to export the data and browse through it in other ways can be easier. I usually use wireshark to browse cleartext web traffic. Now I can use it for encrypted HTTP traffic too. The key is to use mitmproxy along with wireshark’s built-in SSL decryption. Here are the steps to get it working:

Install mitmproxy and get it up and running (We’ll use port 8119 throughout the example here)
Convert the mitmproxy private key you generated into PEM format
- openssl rsa -in mitmproxy-ca.pem -outform PEM -out mitmproxy-ca.key
Setup wireshark to use the mitmproxy-ca.key to decrypt traffic
Start mitmproxy on port 8119
- mitmproxy -p 8119
Start wireshark capturing on the lo interface and filter on port 8119
- wireshark -i lo -f "port 8119" -k
Analyze -> Decode As… -> SSL

You should see the HTTPS traffic in wireshark being automatically decrypted so you can see the layers: TCP, SSL, HTTP.

More Internets!

2014-08-23T00:00:00-05:00

In Austin, there are basically two ISPs, Time Warner and AT&T. That’s not really enough competition to be good for consumers but that discussion is for another time. The lesser of these two evils is Time Warner so that’s my ISP. The other week Time Warner sent me this:

Triple the speed for no additional cost Sounds great, right? Maybe it’s not as great as it sounds. You see, the Internet is not a dedicated line between your house and Netflix. It’s a packet switched network with many varying hops between you and House of Cards. So while speedtest.net may show you rocketing across the internet:

…Netflix still may be limited to 720p instead of the glorious 1080p that your connection warrants. That’s because major backbone providers connect to each other through peering points and if there’s congestion at the peering point between Time Warner and Netflix’s ISP, then Kevin Spacey doesn’t look as crytal clear as he could.

The solution is to rant about these things in public until the ISPs play nice and make their peering points fatter.

People Not Tech

2014-08-22T00:00:00-05:00

My new gig has forced me to grow as an individual and a software developer. When I interviewed I let them know that I had never managed a team before. They said that I “would be fine.” Needless to say it’s been a bumpy ride. Over the last 9 months I’ve come to realize that software in general is less about tech and more about people. I think it partially stems from the fact that software development is more art than science. And software engineers are closer to artists than engineers.

Tale of 1000 Androids

2014-07-26T00:00:00-05:00

Two years ago my trusty Motorola Atrix 4G fell to the pavement(ahem I dropped it) and the camera stopped working. So I went down to the AT&T store since it was time for an upgrade anyway, and bought a Galaxy S3. I ran with the stock firmware for about 30 seconds before rooting, unlocking the bootloader and flashing CyanogenMod. I really like stock Android in place of Samsung Touchwiz. I’ve been running happily with my hacked S3 for 2 years. During that time, my daughter had been using my old Atrix running CyanogenMod. The Atrix finally gave up the ghost so naturally… she gets my S3 and I get a new phone!

I went down to the AT&T store to do the upgrade dance and bought an S4. The S5 is so new that it doesn’t have good CyanogenMod support yet. After monkeying with the S4 for about 24 hours, I realize that AT&T and Samsung have locked the bootloader using mathematically strong, well implemented crypto. So I took it back. No matter how fantastic the phone is, I really want to be able to switch out the ROM. I traded it for a Moto X. I’ve heard good things including it’s low power mode. Plus there’s a page on the Motorola site specifically for requesting a bootloader unlock code! That is, except for AT&T phones. Whoops. Motorola will unlock other Moto X’s but not AT&T’s. I can only assume this is due to some contract between the companies.

At this point I’m pretty fed up. After listening to my rant, the guys at the office said, “Dude, just buy a Nexus 5”. A few clicks and I had purchased a Nexus 5 from Google’s website. Now I’m on my way to return the Moto X to AT&T and let them know that good hardware is not enough. I need the freedom to run whatever I want.

IoT Standards Groups

2014-07-21T00:00:00-05:00

Apparently there are now 3 standards bodies for the Internet of Things, AllSeen Alliance, Open Interconnect Consortium, and Thread Group. The only folks talking about protocol is Thread Group. That’s where the magic is. AllSeen is the most useful today because they offer AllJoyn out of the box but nowhere do they specify the protocol. For a standard to take off, you should define the protocol, and offer a reference implementation. That way other folks can implement their own version. Security Now has a little more info.

Lake Travis Data #681

2014-07-18T00:00:00-05:00

Here’s a little API to get the level of Lake Travis. I’m scraping the official Lake Travis website once a day to get the data. You’ll have to sign in using your github account to get an API key.

J.A.R.V.I.S. Milestone

2014-07-10T00:00:00-05:00

I hit a major milestone today in my little Jarvis project. I got Dragon Naturally Speaking to execute arbitrary shell commands though an SSH session. This is pretty much the last link in the chain of awesomeness. Here’s the flow:

I speak the word “hello” into the mic on my earbuds connected to my laptop running Debian
That audio goes into a Windows XP guest running under VirtualBox
Dragon Naturally Speaking running on Windows performs voice recognition on it and turns it into text

That speech is run through a grammar written in Python configured using natlink and Dragonfly

class WelcomeRule(CompoundRule):
    spec = "hello"                  # Spoken form of command.
    def _process_recognition(self, node, extras):   # Callback when command is spoken.
        spd = Text("spd-say -p -20 -r -50 'Welcome home, Sir!'")
        spd.execute()
        enter = Key("enter")
        enter.execute()

That grammar sends text-to-speech shell command:
```
spd-say -p -20 -r -50 'Welcome home, Sir!
```
…to the open Putty window already SSH’d back to the Debian host
The speech dispatcher sends the text to festival which outputs, “Welcome home, Sir!” though my earbuds.

Thanks to Tavis Rudd for the inspiration.

Errant Whitespace Matters

2014-07-09T00:00:00-05:00

Ugh! Why worry about something trivial like extra whitespace at the end of a line!!? Do you have mental problems or something or do you just hate me?

— Software Engineers Just Tryin’ to Get Stuff Done^TM

Confession time. I think asking developers to remove extra whitespace is important. This is a touchy subject but I think I can make a case so here goes.

Reason #1: Quality

Is software engineering science or art? At this point in my life I’m leaning in the art direction(I should definitely flesh out that idea more someday). Like other forms of art, craftsmanship matters. Think about a woodworker building a chair. Yes, the chair needs to not fall over when you sit in it. But it also matters how beautiful the finish is. The point is that details matter. If your shiny new code that implements that sweet new feature includes extra whitespace then you probably didn’t even bother to double check your commit, did you? If you didn’t bother to double check your commit, there’s a good chance that there are other issues besides the whitespace. Face it… you’re one of those artsy fartsy types now. It’s just that you’re medium is bits instead of wood. Take pride in your work.

Reason #2: Slippery Slope

I’ve seen this a bunch:

commit b41c6310ede83493c2202bc2e3757882b1351a28
Author: Damnthe Torpedoes <geterdone@agilecompany.com>
Date:   Wed Jul 2 15:36:19 2014 -0500

    [ticket #42] Switch to using  ball bearings

    It's all ball bearings nowadays

…with a diff that contains the ball bearing switch over…. along with a bunch of formatting changes automatically introduced by Damnthe’s helpful text editor. Those unnecessary changes makes it almost impossible to see the actual code change amongst the hundreds of lines of code that the editor modified. How am I supposed to review code like that?! How much of this extra junk should be allowed in a commit before somebody yells? I say rather than debate the degree, just say “none”.

Conclusion

Listen whiners… errant whitespace is one of those things that’s super easy to check using a script:

git diff-index --check --cached $against --

So just do it. It’ll make you a better developer. Also it’ll make us OCD types twitch just a little bit less.

P.S. I found errant whitespace in this post and fixed it before committing.

My Dream Job

2014-07-03T00:00:00-05:00

I have some idea of what my “dream job” would be like. I love software so it would have something to do with that. I also like working with smart, talented people so there would have to be some of those. Oh yeah, also I don’t like being micro-managed so there should be none of that. Then it hits me… I think I’m pretty close to my dream job now!

I work at a little company called Key Ingredient where we make a recipe web site. But calling Key Ingredient a recipe web site company is like calling Amazon a book store. We’re doing a ton of other interesting things.

So I’m just going to finish it out and make this my dream job. Here’s the plan:

Software
- The company will fund training and conferences for the developers including me. All good devs love to learn about new cool stuff so we’re going to make that happen.
- Meetups and knowledge sharing - devs are encouraged to attend meetups and give talks about what they’re learned.
Smart, talented people. It’s pretty simple here… hire smart people that can get things done.
No micro-management. I don’t want to work in a micro-managed environment so I refuse to micro-manage my folks. I’ve had the great privilege of working for some incredible managers who I barely saw because they stayed out of the way and let me do my thing. Here’s how we’re rolling:
- Languages, libraries, and tools - each development team picks their own. No mandates here.
- Developers get to pick what they work on. If you’re a Javascript expert who wants to try their hand at Python, go for it. If you’re a backend dev who wants to do a little Android, have at it.

Anyway, I’m just 9 months into this journey but it seems to be going well so far. I’ll keep you posted on how my dream job is going.

Easy PDF Templates

2014-07-02T00:00:00-05:00

We have a standard contractor agreement we use at work. It’s pretty cookie cutter but for every contractor certain things needed to be edited. These edits included name of the contractor, address, start date of the contract, etc. After giving up on forms in LibreOffice, I found an elegant solution. It turns out that LibreOffice od* files are just zipped XML. So I created an Open Document Template(odt) file and replaced all the relevant data with placeholders like __CONTRACTOR__ for contractor name. Then I use a simple make file to:

unzip the template
fill in the placeholders using sed
zip it back up
convert it to a PDF

Here’s the Makefile:

all: Independent-Contractor-Agmt.pdf

CONTRACTOR=Foobar Industries
CONTRACTORLOCATION=Austin, TX
CONTRACTPERIOD=3 months
BEGINDATE=$(shell date +"%B %d, %Y")
ENDDATE=$(shell date -d "+$(CONTRACTPERIOD)" +"%B %d, %Y")

%.odt: %.ott
  mkdir $*
  cd $*; unzip ../$<
  sed -i 's/__CONTRACTOR__/$(CONTRACTOR)/g' $*/content.xml
  sed -i 's/__CONTRACTORLOCATION__/$(CONTRACTORLOCATION)/g' $*/content.xml
  sed -i 's/__BEGINDATE__/$(BEGINDATE)/g' $*/content.xml
  sed -i 's/__ENDDATE__/$(ENDDATE)/g' $*/content.xml
  sed -i 's/__CONTRACTPERIOD__/$(CONTRACTPERIOD)/g' $*/content.xml
  cd $*; zip -r ../$@ *
  rm -r $*

%.pdf: %.odt
  unoconv -f pdf $<

clean:
  rm -f *.odt
  rm -f *.pdf

And here’s how I run it from the command line:

make all CONTRACTOR="Contracts R Us" \
CONTRACTORLOCATION="1234 Fantasy Ln, Austin, TX"

There are defaults for all the placeholders in the Makefile and you can override them on the command line. For example, BEGINDATE is the current date by default. I originally tried using m4 instead of sed but it wasn’t able to parse the XML file. This is just another example of doing things the UNIX Way^TM.

Small Accounting Software Sucks

2014-06-30T00:00:00-05:00

All home or small business accounting software sucks. I’ve tried Quicken for home, and Quickbooks for business. I stuck with ledgersmb for a long time with my business but the web interface was pretty old and clunky. Right now I’m playing around with ledger-cli. Have something that you’re happy with? Let me know.

Snow Day

2014-06-29T00:00:00-05:00

Sylvia Nowakowski Day… Sylvia Now Day… S. Now Day… Snow Day!…

In a crazy, unbelievable, beautiful twist, we are parents. God took us from 0 to 100 in a little over 2 years. Here’s the brief history:

January 2012 - Just me and my bride. No desire for kids.
March 2012 - Several long talks and prayer. It turns out my wife and I both want to adopt. After considering private, international and foster care adoption, we decide that foster care was it for us. If you want the details, email me.
May 2012 - We reach out to Pathways, an adoption and foster care services non-profit.
July 2012 - We begin our foster care training classes. What have we gotten ourselves into?
August 2012 - What ages do we want to get licensed to foster? Let’s do ages 5 to 11.
September 2012 - Reconsidering the age range… not sure… why not ages 0 to 17?
October 2012 - We schedule our home study.
March 2013 - We hear about a 16 year old girl from Laredo who wants to be adopted.
April 2013 - We tell our adoption coordinator that we’re interested in learning more more about her.
June 2013 - We talk to Sylvia for the first time over video chat. We head down to Laredo to meet her in person.
July 2013 - Sylvia decides that we’re her new parents! The state places her with us as her foster parents. We take her down to South Padre to my family reunion.
June 2014 - The adoption is finally consummated!

How Is Pelican?

2014-06-28T00:00:00-05:00

I’m sort of jazzed about static site generators right now for some reason. And since I needed to build a developer blog at Key Ingredient, it seemed like a good opportunity. After checking out Jekyll, Hakyll, Hyde Nikola and Pelican I ended up picking Pelican.

I considered Jekyll because it’s got a lot of buzz around it right now which does count for something. Many times buzz equals users which equals features added more quickly. However, I really wanted something in Python in case I ever needed to hack on it(Ruby is not my favorite). Hakyll was also out for the same reason.

So that leaves Hyde, Nikola and Pelican. The Hyde project seems poorly managed with the old site lying around. Also there hasn’t been much progress on the project in a while. Nikola seems great! The documentation is good and it seems mature. For whatever reason it just didn’t click with me.

That leaves Pelican. Pelican just makes sense to me. The quickstart script, multiple build options, and config setting seem like something I would write for myself if I were so inclined. I really like the fact that I can write in markdown, restructured text, or asciidoc even though I’ll stick mostly with markdown since I’m used to it(github and Unfuddle also use markdown). I also like the fact that Pelican uses Jinja for theme templates because we use it for the templates on keyingredient.com.

For the dev blog, I started with the Bootstrap 2 Pelican theme, upgraded it to Bootstrap 3 and added our logo. Here’s the result: …and the code.

Once the Key Ingredient developer blog was up and running I turned my attention to this site. Fattuba had been languishing for years and my boss Harlan gave me the push I needed to overhaul the thing already!

Open Letter to Embedded Hardware Manufacturers

2012-06-13T00:00:00-05:00

Dear Embedded Hardware Manufacturers,

We are embedded Linux developers. This is 2012. What gives?

It’s time embedded hardware manufacturers get with the times. You may be able to fool the suits but you haven’t convinced us. That is, you haven’t convinced us that your semi-walled garden is the way to go. Yes, you offer Linux support in your BSP. But that is no longer enough. You see, we have gotten use to Googling for answers, asking for help on IRC and seeking advice from peers via email. Dial-in support numbers are useless and “live” chats from your website make us feel ill.

We’re tired of hacky build systems that only support your platforms. What does it say about your “Linux support” if your patches aren’t good enough for mainline?

We build our own kernels straight from kernel.org. Support for your platform should be there. We run OpenWRT on our own routers at home. Your platform should be there when I type make menuconfig. If you have “secret sauce” that makes your platform better than the competitors, it better be easy to build, integrate, and use.

In short if you want us to recommend you and your platform to the suits…

Have a great hardware
Get your platform into Linux mainline - Sometimes this can make up for slightly weaker hardware
Get your platform into OpenWRT - Sometimes this can make up for non-mainline Linux support

Sincerely,

Embedded Linux Developers