Cascalog workshop

I'll be teaching a Cascalog workshop on February 19th at BackType HQ in Union Square. You can sign up at http://cascalog.eventbrite.com. Early bird tickets are available until January 31st.

I'm very excited to be teaching this workshop. Cascalog's tight integration with Clojure opens up a world of techniques that no other data processing tool is able to do. Even though I created Cascalog, I've been discovering many of these techniques as I've made use of Cascalog for more and more varied tasks. Along the way, I've tweaked Cascalog so that making use of these techniques would be cleaner and more idiomatic. At this point, after nine months of iteration, Cascalog is a joy to use for even the most complex tasks. I'm excited to impart this knowledge upon others in this workshop.


Analysis of the #LessAmbitiousMovies Twitter Meme

We did a fun post on the BackType blog today analyzing a meme that took off on Twitter this week. A person with about 500 followers started the meme that eventually reached more than 27 million people. Check out our analysis here, and you can check out TechCrunch coverage of our analysis here.

Doing the analysis was relatively simple. We extracted an 80 MB dataset of the tweets involved in the meme from our 25 TB social dataset. We downloaded that data to a local computer and ran queries on the data from a Clojure REPL using Cascalog. Doing the data analysis only took us a couple hours.


How to reject a job candidate without being an asshole

I used to send a job candidate an email like the following after the person failed a phone interview:

Hi [Candidate],

Thanks a lot for taking the time to interview with us. However, we've decided not to move forward in the process with you. I wish you the best of luck on your future projects.


Rejection emails like that are cold, impersonal, and hollow. It doesn't feel good to send an email like that, and it sure doesn't feel good to receive an email like that.

You want a candidate to feel good about your company even after failing the interview process. You don't want the candidate to discourage their friends from interviewing with your company, and ideally you want the candidate to refer their friends to your company.

Now I tell candidates on the spot whether they pass or fail at the end of the phone interview. I give them feedback on what they did well and what they did poorly. I'm very candid with them.

The first time I tried this, I had butterflies in my stomach. I thought to myself, "What if he gets angry? What if he freaks out on me and starts screaming obscenities?" It turns out my fears were completely unfounded.

I've found that candidates really appreciate my candor about their rejection. They appreciate being told why they failed, especially because everyone else is so impersonal and non-specific about rejections.

When I reject a candidate, I lead with what I liked about the person. For example, I might tell the candidate how I thought the project he described was interesting. I find that leading with an honest compliment softens the blow of the impending criticism. I haven't yet been at a loss for something nice to say to a candidate, and it would be a rare candidate for that to happen.

Next, I tell the candidate why we can't move forward in the process with them. I tell the candidate what we're looking for and why I think they're not qualified. Let's face it: candidates know when they didn't do a good job. They know if they bombed the coding question you gave them. I tell the candidate that I know it's possible that we're making a mistake, that coding questions aren't always the most accurate indicators, but that it's the decision we have to make based on the results of the interview.

Finally, I give candidates a chance to give me feedback. I ask what they thought about our interview process, and I ask what I could do better as an interviewer. I've gotten some valuable feedback this way, and I'm becoming a better interviewer because of the feedback. Plus, giving candidates the opportunity to criticize me makes the process feel a lot less one-sided.

I'm really happy about my decision to reject candidates on the spot at the end of phone interviews rather than with impersonal emails. I don't feel like an asshole anymore, and candidates seem to appreciate the feedback.

You should follow me on Twitter here.


You Are a Product

I had a revelation the other day. I realized that the terms "programmer" and "employee" are inadequate to describe what I am. What I am is a product, and you are one too. If you want to develop your career, you need to approach your career as a product development problem.

You sell yourself for various things: money, status, the opportunity to work on interesting problems, good coworkers, etc. In this post I'll be referring to this as "getting paid", but please keep in mind that "getting paid" means more than just money.

Supply and Demand

Like any product, you have supply and demand. Your supply is what you can do for a company that hires you. It's your ability to make beautiful websites. It's your ability to scale a database. It's your ability to get the best work out of others. Your supply is the actual value you will provide to a company that hires you.

Your demand is what companies think you can do for them. Your demand is your perceived value by others. At the end of the day, you will be paid according to how you're perceived, not by the actual value you can produce. This is why so many 10x engineers don't actually get paid 10x -- they're not publicly perceived as 10x engineers so normal market forces are unable to bid up their value.

I see way too many people say to themselves "As long as I just put out quality work, I'll be taken care of." This is bullshit. This way of thinking prevents you from reaching your potential. It prevents you from being paid what you should be getting paid, and it prevents you from bettering your status. You cannot just focus on your supply. Supply is only 50% of the equation. You could be the greatest programmer to ever live, but if no one knows that, it won't help you. You are a product, and if you want to get paid appropriately, you have to work on your demand.

Personal branding

Influencing your demand is called "personal branding." It's marketing. Your actual value -- your supply -- is important to the extent that you can use it to raise your perceived value -- your demand.

Personal branding is inherently a public activity. Market forces rely on information being public. You want lots of people to believe that you can provide them with lots of value. This will lead to opportunities for you. Many of these opportunities will be out of the blue and unexpected.

There are lots of things you can do to increase your demand. Start a blog and promote it through Twitter and social news sites. Speak at conferences. Build social proof by building up your Twitter follower list. Participate in open source projects and write blog posts about the work you're doing on the projects.

I think open source is the best activity a programmer can engage in. It makes public your actual ability to solve problems and write code. You should strongly prefer to work at companies with a culture of making and contributing to open source projects, as that gives you the opportunity to market yourself.

I think the best personal branding activities are rooted in the actual value you can provide to others. There are other activities that can increase your demand, like taking credit for the work of others, that are flat out unethical. Don't be a product that's just smoke and mirrors.

Marketing yourself takes work, but it's something that gets easier with practice. It would be stupid to release a product to market without promoting and marketing it. Likewise, you should treat yourself as a product and market yourself as such. When you do, you can watch the forces of supply and demand work their magic.

You should follow me on Twitter here.


The time I hacked my high school

When I was in high school, I started the Chess Club. I needed money to buy chess sets and chess clocks to get the club going, but at first I had some difficulty raising cash.

Then I hacked the system, and Chess Club became a cash generating machine.

Before the hack

Clubs made money by reselling burritos or pizza from nearby restaurants during lunch. Each of these lunch sales typically made about $100 in profit. Since I didn't want to charge dues for the club, I needed lunch sales to raise money for Chess Club.

Unfortunately, the rules around lunch sales were restrictive. Only one club could sell per week, and other clubs like the Science Club had a much stronger precedent for needing lunch sales. Without a precedent for needing money, I was unable to acquire enough lunch sale dates.

The hack

I studied the rules for operating clubs on campus and found the loophole I needed: clubs were allowed to go into debt to the student government for $200. I figured that if I were in debt to the student government, I'd have more leverage in getting lunch sale dates.

I immediately spent $200 on chess boards, chess clocks, and books. I bought more than we needed because I wanted to maximize our debt. Then I went to the student government treasurer, gave him the receipt, and was reimbursed for the expense.

The student government wasn't too happy about the situation. They wanted me to pay them back as they were on a tight budget. I told them I couldn't raise money because they wouldn't give me lunch sale dates.


They relented and started giving me lunch sale dates so that I could pay them back. Even though we made $100 per lunch sale, I only paid them back $50 at a time to maximize the time we were in debt. Soon afterwards, the student government relaxed the rules to let clubs have lunch sales more days per week.

Since Chess Club now had a precedent for needing money, I was able to get plenty of slots for lunch sales. The student government clearly didn't think about why we needed so much money (we didn't need so much money). They relied on the "precedent for needing money" heuristic and may have been afraid Chess Club would go into debt again.

After the hack

Chess Club was swimming in money. I stocked the school's library with chess sets. I upgraded the club's chess sets to a mixture of glass and wooden sets. I bought computerized chess sets and expanded our collection of chess books. I started holding school-wide chess tournaments with hundreds of dollars in prizes.

I couldn't spend money faster than we were bringing it in.

When I graduated, Chess Club had $500 in the bank. I considered holding one last tournament with massive prizes, but ultimately decided to leave the money for future generations of Chess Club.

You should follow me on Twitter here.


Fastest Viable Product: Investing in Speed at a Startup

A startup is like a rat in a maze searching for a piece of cheese. The cheese in the startup's case is product-market fit, that pivotal point when the startup can scale and monetize the business.

In the maze, the startup has a dazzling amount of choices of where to go. Should we build this new feature? Should we try this new idea we have for a product? Should we backtrack and completely change our idea?

Lean startups use a strategy called "Minimum Viable Products" to help navigate the maze. The idea is that a startup formulates hypotheses about what users want or do not want; each of these hypotheses is a "turn" in the maze. A "Minimum Viable Product" is the smallest test that will let the startup know whether their "turn" was a good one. A startup wants to stop going the wrong direction as early as possible.

A "Minimum Viable Product" can be anything from a working application to an SEO'd survey that will gauge interest in an idea. "Minimum Viable Products" have been written about extensively.

The term "Minimum Viable Product" is a misnomer

However, the term "Minimum Viable Product" is a misnomer. The real goal is to test hypotheses as fast as possible, and being minimal is just a side effect of being fast. "Fastest Viable Product" is a more appropriate name.

Click to read more ...


How to get a job at a kick-ass startup (for programmers)

When I finished college, I was incredibly naive when it came to finding a great job. I knew that I wanted to work at a small startup but didn't know how to find that great opportunity. I didn't know what questions to ask to evaluate a company, and I didn't know how I should present myself during the recruitment process.

Now I'm a few years out of college and I have that kick-ass job I was looking for. My dual experiences of looking for a job and being on the other side recruiting programmers have taught me quite a bit about what it takes to get a great job at a kick-ass startup.

Here are my tips, from preparing for the job search process to finding great startups to applying and getting the job. If you have any tips of your own, be sure to leave them in the comments!

Click to read more ...


Clojure or: How I Learned to Stop Worrying and Love the Parentheses

I'm a longtime Java, Ruby, and Python programmer. Yet Clojure is the first language I've used that I truly enjoy using on a daily basis.

Clojure is a special language. There have been many attempts to articulate the benefits of Lisp-based languages before, but most of these attempts seem to end in futility. Until you use the language, it's hard to understand why functional programming, macros, and immutability are such a big deal.

So I'm going to take a different approach in explaining the virtues of Clojure. I'm going to start off somewhat unusually by talking about SQL, show how querying is done fundamentally differently in Clojure, and transition from there into a broader discussion about domain specific languages, accidental complexity, and how Clojure solves problems that have plagued programmers throughout programming's history.

The problem with SQL

SQL is a language for querying relational databases. SQL is one of the most successful technologies ever, but very few technologies can claim to have led to as much unnecessary complexity as SQL.

SQL solves the problem of querying a relational database in a concise, expressive manner. In that regard, SQL does a very good job.

The problem with SQL is that it's a custom language. Using SQL from other languages causes a host of other problems - problems that are orthogonal to querying a database. These problems are examples of accidental complexity, complexity in an application caused by the tool used to solve a problem rather than the problem itself.

The prime example of accidental complexity caused by the nature of SQL being a custom language are SQL injection attacks.

SQL injection has nothing to do with querying databases

SQL injection results from using one language from within another by doing string manipulation. This has nothing to do with querying databases, it's an integration problem. As we'll see later, it's a problem we can avoid in Clojure.

Click to read more ...


5 Tips for Thinking Under Uncertainty

Most people flee uncertainty. Yet being able to think well under uncertainty can be very rewarding, and I've come to realize it's a skill that can be learned. Here are some tips for thinking under uncertainty.

Click to read more ...


You should blog even if you have no readers

Spencer Fry wrote a great post on "Why entrepreneurs should write." I would further add that the benefits of writing are so extraordinary that you should write a blog even if you have no readers (and regardless of whether you're an entrepreneur).

I have over 50 unfinished drafts. Some of them are just a few ideas scribbled down arguing with myself. Most of them will never be published, yet I got value out of writing all of them.

Writing makes you a better reader

Blogging has changed how I read other people's writing.

In struggling to find the right ways to structure and present my posts, I am much more attuned to what makes a good argument and what makes a bad argument. I am better at seeing holes in other people's reasoning.

At the same time, when reading I am less likely to fall into the trap of discrediting a post with weak counterclaims. In most any post, there are likely to be counterclaims that are based on exceptional cases. Internet commenters love to point these out. However, these exceptional cases miss the main thrust of the post, and by understanding the implicit backdrop behind a post's argument, I get a lot more value out of reading.

I'm also more aware of the style of good writers. I mentally take note of the ways good writers phrase their ideas. I'd always enjoyed Paul Graham's writing, but now I really appreciate how he organizes his posts. He has an awesome ability to suck you into his world and show you what it looks like from his perspective. I've learned a lot about good writing from reading Bradford Cross's blog; his posts have a clear arc and make excellent use of short paragraphs to keep the posts flowing.

Writing makes you smarter

Writing reveals holes in your thinking. When your ideas are written and looking back at you, they're a lot less convincing than when they're just in your head. Writing forces you to mature your ideas by thinking through counterarguments.

Writing helps you organize your thoughts in a coherent way. This makes you a much better conversationalist when these topics come up. I can't count the number of times I've had deeper conversations with people because I had matured my ideas offline.

Consider anything else a side benefit

Everything else writing gives you -- personal branding, networking, inbound opportunities -- are just side benefits. They're potentially very large side benefits, but they are not the main reason you should write.

You should write because writing makes you a better person.

You should follow me on Twitter here.