Adam Smith's Blog

How to: Compete with Amazon S3 without Buying Hardware

You build a storage router.

Given a piece of customer data you want to store, replicate it 2x, and store half a copy's worth across four different raw storage providers.  For example, Rackspace gets one half a copy, as does SoftLayer, GoGrid, and Storm on Demand.  Keep in mind that you're buying raw storage from each of these guys, not redudant storage.  Each provider is just selling you hard drives attached to the Internet.

Your customer's data will be available as long as two of the four raw storage providers are up.

(To be clear, you would use Reed-Solomon erasure codes.  Thus, you could dial in any desired replication factor and any desired number of storage endpoints. [1]  The blocks in the diagram below are intended to represent the sizes of the data pieces, not how they are encoded.)

 

This design could provide more independent failure modes than Amazon S3.

More importantly, this design would create a marketplace for people selling raw storage.  Whenever someone comes in with a better price, you seamlessly move customer data from the most expensive of your providers to this newer, cheaper one.  Of course you would maintain appropriate quality controls [2], monitor uptimes, backbone redundancy, etc.  You're a storage router.

 

Amazon S3 enjoys high margins today

Amazon's storage offering has built up high margins.  Up until two months ago, Amazon S3 hadn't dropped prices since its launch in 2006, despite the fact that hard drive costs fall 50% per year. [3]

In 2006 a 320 GB hard drive cost $120.  Today (Thailand floods aside) that much money will snag you a 3 TB drive.

Those are real margins that Amazon has built up, and even after their recent price drop they're still leveraging a lot of pricing power.  There's surprisingly little competition.

 

The Android strategy for the cloud storage market

Just as cell phone OEM's like Samsung and HTC didn't know how to compete with the iPhone, the guys left in the dust by Amazon (Rackspace, SoftLayer, and moreso the hundreds of small hosting companies out there) don't know how to compete.

What I'm proposing is the Android strategy because it gives them a way to compete, and they're competing on an axis that the new winner doesn't want to compete on: price.

And in this case, since you're running the service that routes customer data into and out of the right storage providers, you control the relationship with the customer, so you get the pricing power.

 

The challenges

There are a couple of reasons why I'm not pursuing this idea (in increasing order of importance):

• Selling storage requires a brand that people trust.  That could be built, but is a very big friction point at the most inopportune moment - the beginning.
• It's a low margin business, and you'd probably need a sales team eventually.  Both of those are unfortunate.
• Amazon Web Services has revenue from a number of complimentary services (most importantly EC2) that they could use to make storage a loss leader, sucking all of the oxygen out of the room.  Amazon did this in 2010 to diapers.com.  They know how to run and compete in low margin businesses.  Amazon would also compete by giving deep discounts to large customers who suggest they might use this service instead.  Amazon's dominance and ecosystem is the most important weakness of this idea.

 

Who then?

The world would be a better place with a service like this, possibly extending into a marketplace for compute as well.  Someone like Rackspace or Softlayer could do it.  Google could do it.  A startup could do it.  Or maybe OpenStack will adopt a model like this.

If this happens, though, I think it will be done by a startup.

 

What Rackspace, SoftLayer, and others should do now

The big hosting guys should launch a low priced "hard drive attached to the Internet" product. [4]  Such a move would pave the way for one or more startups to build the storage router, bringing a new type of innovation to cloud computing.

 

Notes

[1] The replication doesn't need to be uniform, either.  For example, if read performance is important maybe one storage provider has a full copy of the original data, and there's another 1.5 copies spread across three other providers for redundancy.  The ideal setup depends on the customer's desired tradeoffs.  This advanced level of control might also be a competitive advantage for some storage customers.

[2] There are many types of quality controls to build.  Use a provable data possession protocol to ensure that each storage provider still has its data without having to retrieve the whole thing.

[3] This is strictly true for their first tier of storage customers.  Over time they've published volume pricing for higher and higher volumes, though each tier's pricing never dropped.  Amazon S3 pricing history: 2006, 2007, 2008, 2009, 2010.

[4] Such a product should be designed for the storage router use case, so it should include all of the API's the storage router will need, including provable data possession.

 

Grit > Self Control > IQ

When I was at MIT all computer science majors were required to take a class called 6.001.  I had a lot of college credit coming to MIT so I was able to place out of many other requirements, including chemistry, biology, and calculus.  I also had a long history of programming experience, so I wanted to skip 6.001.

This was impossible, I was told.  Nobody had ever done it.

In a saga that unfolded over the next two months, we negotiated a compromise agreement.

None of that would have been possible if I wasn’t willing to make a (polite) pain in the ass out of myself.

The world has a way of making room for those who just won’t give up.

This isn’t a new concept in startup land.  Paul Graham, for example, talks constantly about determination.

So rather than turn this into another post about how entrepreneurs these days need more of X, I wanted to share some scientific research that studies determination, and brainstorm on how we can boost our determination.

Angela Duckworth, from UPenn, has published much about the topic.  (Obligatory TED video here.)

She uses the word grit, which I like.

She defines grit as “perseverance and passion for long-term goals.”  That sounds about right.  She continues: “Grit entails working strenuously toward challenges, maintaining effort and interest over years despite failure, adversity, and plateaus in progress.”

Ok.  How is grit different from self-control?

Grit is also distinct from […] self-control, in its specification of consistent goals and interests. An individual high in self-control but moderate in grit may, for example, effectively control his or her temper, stick to his or her diet, and resist the urge to surf the Internet at work—yet switch careers [frequently]. As Galton (1892) suggested, abiding commitment to a particular vocation [..] does not derive from overriding “hourly temptations.” –Grit: Perseverance and Passion for Long-Term Goals, page 1089

Ok, so self-control is about not eating the marshmallow right in front of you, and grit is about pushing towards the same goal for long periods of time, even if you don’t get short term results.

How do grit, self-control, and IQ predict success in various efforts?  Dr. Duckworth has done several studies.  Let’s look at the results of a spelling bee competition.  That shouldn't differ too much from entrepreneurship!

 

According to this figure, if you could choose to be in the top quartile of participants for grit, self-control, or IQ, you would pick them in that order.  Grit > self control > IQ.

That’s not really surprising to me as an entrepreneur, but it’s validating since the world seems to think that IQ points win spelling bees.

Why might most people think that IQ and self-control are more important than consistent, long term effort?  There are many possible answers, but Duckworth’s research suggests an explanation I hadn’t thought of before: that it’s harder to observe long term focused efforts by others.

We speculate that individual differences in the intensity dimension of effort are salient and, therefore, described by many adjectives in the English language (e.g., energetic, conscientious, dutiful, responsible, lazy). Whereas the amount of energy one invests in a particular task at a given moment in time is readily apparent both to oneself and to others, the consistency of one’s long-term goals and the stamina with which one pursues those goals over years may be less obvious. Similarly, whereas the importance of working harder is easily apprehended, the importance of working longer without switching objectives may be less perceptible.

Ok, so how do we train ourselves to be grittier?

That’s not clear.  Dr. Duckworth is doing research on this question now, so perhaps we’ll have a follow up post in a few years (follow me on twitter here!), but until then here are some thoughts:

• First, Duckworth cites a study that identified a strong interest in one’s particular field as an ingredient for high achievement.  Steve Jobs pointed this out in his Stanford graduation speech where he notes that the only reason he recovered from his failures was his love for what he did.  So find what you love, and grittiness will come more naturally.
• Second, I wonder if it’s possible to trade away some self-control to get more grit.  That sounds like a good bargain.
  
  Some studies show that willpower is a limited resource; if you use willpower for task A then you’ll have less left over for task B.  If this is true, maybe you should exert less self-control in some areas in order to have more willpower left over for commitment to your long term goals.
  
  I should note that this is highly speculative.
• In a related fashion, I think grit is diminished when you work too hard and burn yourself out.  I’ve seen startups where everyone worked really hard, got early traction, and then sold their company too early because they ran out of energy for the marathon.
• Finally, like many other things in life, I’m guessing you can increase your grit by surrounding yourself with gritty people, especially those with matching long term interests.  If that’s true, you should find gritty cofounders, invest in gritty companies, and surround yourself with gritty entrepreneurial friends.

What do you think?  How can we increase our grittiness and bring it out in others?

(As a homework exercise, consider what you want to be gritty about.  Steve Jobs suggested being gritty about finding work you love, doing great work, and finding a great spouse.  That's a really good start!)

(Thanks to Zao Yang for turning me onto Dr. Duckworth's work!)

Announcing: DesktopBootstrap

DesktopBootstrap is like Twitter Bootstrap, but for desktop apps.

It's the open source project I got to work on over Thanksgiving.

If you've ever written a desktop application before, you probably spent several weeks working on the auto update, installer, and logging mechanisms.  Hopefully this will make your next one that much easier!

Towards Better Online Dating

It’s clear that the existing online dating sites are all broken.

I thought about working in this startup space for a few months last year, so I wanted to document my thoughts on the subject.

I ended up deciding against working in this space for now, but I still find it highly interesting and know that for the right kind of entrepreneur it could be a great place to dig in.

As you’ll see, the right kind of entrepreneur for this space is one who is willing to iterate and try lots of things, who understands human interactions and economics on a deep level, and who has the patience of a saint.

A few introductory thoughts to get us started:

1. Meeting friends-of-friends is the best way to find cool people, because liking is transitive AND because the low social distances force people to be on their best behavior.
2. Online dating is on the OTHER end of the spectrum.  Social distances are incredibly high.  99% of the time your dating profile doesn’t reveal your real identity, or real friends, so there is no disincentive to behaving antisocially.
3. This ‘behaving antisocially’ manifests itself in multiple ways, namely spamming, hypergamy, and false advertising.  Let’s talk about each of those.
4. Spamming is the act of sending messages to tons of people.  The more messages you send the higher your chances of getting dates, but that also lowers the efficiency of the site.  It’s a greedy individual behavior that hurts systematic efficiency.
5. Hypergamy is the act of pursuing people who are higher social value than oneself.  Dorky Dillon wouldn’t have any shot at getting with Model Mary, but he won’t hesitate to lob a message over in high social distance situations where there’s no personal downside, risk of public rejection, etc.  Again, this is an individual behavior that hurts the broader system.
6. Finally, false advertising is embodied in the “MySpace Pose,” the overly-witty profile, pictures of me on my BMX bike (ha!), etc.
7. It’s important to understand that you can’t blame the users for any of these things.  The users are just responding to the incentives created by the modern online dating site.

My favorite approach to the online dating problem is to lower the social distances.  But before we get into that, let’s talk about the other major challenge: network effects.  Also known as the chicken and egg problem.

I’ll leave the network effect problem as an exercise to the reader, and refer you to this Chris Dixon blog post, specifically the part about irregular network topologies, and this slide deck RE getting lots of users.

Aside from the network effects encouraging incumbent laziness (a la eBay), there are three other reasons why the status quo in this space is quite poor:

1. The big players are making lots of money off of their self-destructive business model.  We’ll talk more later about why their business model kills long term value, but suffice it to say that they’re not looking for ways to grow the market that involve cannibalizing their revenues.
2. Second, there are an astronomical number of new approaches a new entrepreneur could try.  The search space is quite large so most people throw their hands up in the air and copy large chunks of broken DNA from existing approaches.
3. Finally, when new entrants do try new approaches, most of the time they aren’t well contemplated.

But there is still a ton of opportunity.  In the words of my respected friend Zao Yang (FarmVille creator), “Online dating is like the mobile world before the iPhone.”

That quote hits the nail on the head, both in terms of the magnitude of the opportunity and in terms of how hard it is to do really great work in this space.

And to redefine the problem a little bit, I suspect there’s more value and fun in helping people meet new friends generically, and only incidentally maybe a significant other.  I only talk about this problem in terms of ‘online dating’ because that’s the existing anchor in peoples’ minds.

So let’s say you’re crazy enough to take a shot at this unicorn.  How should you think about winning this market?

I think of dating sites as the sum of three components: the business model, the back end, and the front end.

The status quo sites like Match and eHarmony have all three components wrong, I believe.

Let’s talk about the business model first.

 

Business models

The most profitable sites charge users for the ability to send messages to other people.  While this scheme generates $350M in annual revenue for Match, it kills long term retention, word of mouth, and the overall user experience.

The average user lifetime on Match is abysmally low.  I can’t remember the exact number I heard from industry insiders, but it was about six months, and maybe shorter.

Imagine if Facebook had a user lifetime of six months!!!!!!!!

(I realize that once there’s a successful match the couple leaves the site, which is another reason such a site would be better positioned as a way to meet new people in general.  But I’d wager that most people leave because they didn't find a significant other via Match.)

Anyway, here’s how it unfolds.  I sign up on the site, and buy a subscription.  I find six or seven girls I like, and send them highly personalized messages.  About 15% of people on Match are premium members, and therefore only one of those girls can even reply to my message.  I get one reply, at most.

So then I start spamming, because I have to, and I'm incentivized to.

…It gets worse.

It’s also in Match’s best interest for me to send spam to a bunch of non-paying members, because Match builds revenue when new users convert because they’ve received a message they can't reply to!  It’s for that reason that Match doesn’t tell you who’s a premium member and who’s not.

This broken business model works for big sites that have the network effect lock in, and niche sites that serve particularly affluent markets, like Jdate.

Yet I predict those sites will all die when someone really figures out the online dating puzzle in the next decade.  And it won’t just be an innovation in business model; it will come from changes in all three components of how dating sites work.

OkCupid wrote about this business model problem extensively on their blog, and I learned most of this from talking to those guys.  (Update: since I originally wrote this OkCupid got bought by Match and took down the blog post I wanted to link to.  It’s called Why You Should Never Pay For Online Dating, and Google Cache has a copy here.)

 

The Backend

The backend to me is everything that calculates matches, who can see who, etc.

The backend is the only of the three components that I could nail if I were working on online dating.  It’s right up a data engineer’s alley.

It’s no surprise that 18 year-old Match takes a simplistic approach to everything.  As far as I can tell, every user can see every other user, there’s no opaque throttling down of a spammers’ message delivery, and so on.

These sites also don’t build real signal into people’s dating profiles.  A girl can learn more in five seconds of "real life" interaction than from reading a guy's OkCupid profile.

A bad backend?

…No…a horrible backend.

Here’s the principled way to approach this problem: close the loop and get real data about what kind of matches work and which don’t.  eHarmony claims to do this but they seem to employ PhD’s in “love science” rather than statisticians.  Call me cynical but I’ll take the statisticians, thanks.

So you run a speed dating event with random people.  Get them to submit a copy of their Facebook profiles.  Run computer vision algorithms on their Facebook photos to see how much skin they show in photos, how often they’re smiling, how many people they’re in pictures with, and so on.  Extract a lot of signal from some source of ground truth.

Then watch which people are interested in each other after the speed dating event, and use off the shelf algorithms to build a classifier that will, given two Facebook profiles, predict the probability that any two people would enjoy four minutes of real life interaction.

Then only allow people to access others they are likely to match against.

You should also watch this talk by Joel Spolsky about the sociology that goes into Stack Overflow, if only as an example of the flavor of thinking you need to do.

OkCupid seems to be at the forefront in this space.  They get people in India to rate the attractiveness of new members and use that to limit your access to people significantly more attractive than you are.  That’s not a principled approach but a step in the right direction compared to other incumbents.

You could really spend years optimizing the crap out of this.  And it would be worthwhile.

 

The Frontend

This one is the most nuanced.  Most of it has to do with behavioral psychology and how people interact with technology.

The front end is everything that makes up the product that isn’t the backend or the business model.  It’s the positioning, the user flows, and so on.

First, I’m not convinced that positioning oneself as a dating site is optimal.  I like to think of the problem as Helping People Meet Cool People.  Not only does this make a difference in positioning/stigma, it feels more wholesome, more valuable, and more practical.

Secondly, everyone knows that dating profiles suck to create and suck to consume, as mentioned above.  Not only do they lack real signal, they also make for a horrible onboarding experience.  You want me to fill eight huge text areas with witty banter, and check or uncheck 250 radio buttons?

Geesh.

Overall I’m not bullish on the trinity of people profiles, messaging, and people searches.  I know that’s how all sites are built today, but if I were in this space I’d keep an open mind on redefining the primitives.

For example, what about a site that just schedules weekly group dinners with new people?  Every Wednesday night you’re set up for dinner with five other people we think you’ll enjoy.  Not only would this get you past the high social distance problem, it moves the online posturing-and-judging step into the real world where it probably belongs.

That particular idea might not work, but I stand by its disruptive nature.

(Since originally writing this I’ve learned about Grubwith.us.  Fantastic!)

I would also be excited about anything that approaches the problem circuitously.  Imagine if you had a mobile product which helped friends hang out with their existing friends.  And then you added the ability to meet new people you might like.  And so on.

At the end of the day, whoever figures this out is more likely to stumble into the solution rather than foreseeing the right ecosystem in a monolithic, Genesis 1:1 fashion.

 

Overall, the world will be a better place once this space is revamped.  Meeting people is a huge part of life.  Today it's just too O(n).  Motive, and opportunity.

 

(Update, 1 year later: One year later, many new ideas have been tried, with limited success so far.  One of the prodomenant themes is connecting users to people closer to you on the social graph, for example with Facebook integration.  This kind of targeting is akin to what I discussed in the 'Backend' section, and is a step forward.  It's largely enabled by broader social acceptance of online dating displacing the need for pseudonym-based online dating.  But the larger, harder questions -- how to overcome adverse selection bias, and the bootstrapping problem -- remain insoluble by all of the approaches I've heard about lately.  For example, at the present day, if your user onboarding process begins with "go to xyz.com," I think you've already generated a lot of selection bias; you'll be missing out on a whole segment of the population that would otherwise add a lot of value to your ecosystem.  I also haven't seen any significant advances in solving the bootstrapping problem.

One of the more interesting approaches I saw was at the Startup School office hours session in 2011 where one of the companies was proposing building better tools for matchmakers.  If you ONLY target the matchmakers as users of your website you might be able to avoid the selection bias and bootstrapping problems.  I think it remains to be seen whether that approach has other problems, for example whether matchmakers are incentivized enough to risk their energy and social capital to try to make things happen.)

(Update 2: I think there might be a business in building backend matching software and data sets, and selling it to every dating site.  There are so many folks building products that are intended to match people, and they all lack any measure of sophistication in how they do it.  Their innovations are usually in the front end; they could certainly use help with their matching backends.  You could achieve network effects by accumulating training data from a variety of sites, by requiring that they send you feedback data regarding how any given match turned out.)

 

Idea: Y Combinator for Startup Recruiting

Recruiting more college students into startups is a constant topic of discussion.  Just today Fred Wilson mentioned it as one of the critical ingredients for the NYC community.

It always strikes me that more undergraduate hackers go to Wall Street than the startup community.

Maybe we can do something about it, and that's what I want to explore in this blog post...

Y Combinator is a great hack on the world.  It drives $billions of value creation by investing super small amounts.  It’s the ultimate in leverage.  Could college recruiting for startups be similarly hackable?

Well, why do so many students go to Google and Wall Street rather than startups?  It's a complex issue and there are lots of reasons, but here are some of the top ones:

• College students are irrationally risk averse. Instead of paying annual salaries and bonuses, startup upside is captured in option grants that are riskier.  Many of the best college students at MIT are risk averse and think short term. These dispositions are highly cultural.
• Prestige. Startups are missing brand value for parents, relatives, and friends.  This is also highly cultural.
• Too Many Startups / Choice Paradox. Students have no idea how to pick the right startups.  Even if they did, there are an overwhelming number of choices.  This is classic paradox of choice.
• Mismatched Timelines. Startups are often too short-sighted to do college recruiting well, since it takes a long term investment strategy.

Frankly there’s a lot of content and blogs out there about how to start your own company.  And Y Combinator has come around.  That is all very awesome.

But there’s so little content out there about what it’s like to join a startup, or how to join a startup right out of college, etc.

We’re not marketing to our customers very well.

And more broadly there are no concerted efforts to work on this problem.  What could we be doing together to move the needle for the whole community?

We wouldn't need the entire valley to cooperate here; you could do some interesting things just with just a few startups or as a single VC fund.  Here’s what you would do:

1. Start by focusing on getting great interns for startups, and move to full time recruiting later as you learn and iterate.  (Successful internships lead to full time hires anyway.)
   
   
2. Set up an open door application that any student can apply to, much like Y Combinator.  Also like Y Combinator and Startup School, the application should ask high signal, aggressive questions.  Resumes are required but of secondary importance.
   
   
3. Find the applications that you think are promising.  The startups involved will get to browse through the remaining applications to decide who they’re interested in.
   
   
4. Select a very particular breed of startups to participate – specifically startups with meaty internship projects where the company has some momentum and has already shipped their first product.  (The students are the customers here, not the companies.  The long term health of the program will depend mostly on quality of the students’ experiences.)
   
   
5. Try to give students a list of three startups that are interested in them.
   
   
6. If there’s only one startup interested in a student, it probably means you didn’t filter applications heavily enough before having the startups look at them.
   
   
7. If more than three startups want a student I would call that student and coach them through deciding which three startups are most interesting to them.
   
   
8. Fly the student out to meet and interview with each of the three startups.  At this point the student and the startup will already know a lot about each other, so the interviews will be two-way and probably more focused on evaluating culture fit.  Give the students some ideas on how to identify the best startup for them.
   
   
9. While we’re at it, throw in Y Combinator style weekly dinners for the interns, with speakers from the startup ecosystem.

Word of mouth would be hugely important to the long term success of such a program.  Corollaries:

• It'd be helpful to have a brand to leverage from the beginning.  Y Combinator could do it.  Sequoia could do it.  If someone without a brand were to do this they could try to partner with one of these guys.
  
  
• Start with a medium push.  Initial momentum will be important, but so will your long term endurance.
  
  
• Build an alumni network of the students who have been through the program.  They will help each other for years to come.
  
  
• Leverage your alumni network to build awareness at top CS colleges.
  
  
• Make sure the students are blown away every step along the way.  Customer service baby!
  
  
• Be picky about who you accept when it comes to students AND startups.  Pairing a desperate company with a desperate applicant would dilute your brand and wouldn't be worth your time anyway.

So why is this important?

Startups deserve to win the battle for top CS graduates.  These hackers are young; they should be taking risks early in their career.  Their talent would be more leveraged at startups.  They would learn less about bureaucracy and more about creating products and value.

But wait!  It gets better!  …working at a startup can also prepare people for starting their own companies down the road, if they so choose.

Case in point: a few years ago at Xobni we recruited a killer engineer for an internship.  He came to work at Xobni as a full time employee a year later.  After doing some more fantastic work at Xobni, he is now starting his own company, and I think that’s a wonderful cycle to feed.

Adam’s Life Hacks Part Two: The Lean Paper Filing System

“A filing system is absolutely essential.”

–Randy Pausch, Time Management

 

Unfortunately everyone has to deal with some number of physical papers, CDs, and so on.  Enter the personal filing system.

Most filing systems fail out of the gate because they don’t optimize for insert time.  Since starting my filing system two years ago I’ve organized 112 documents, and in those two years I’ve probably only retrieved a document five or ten times.

So here’s what I would suggest:

1. Grab a box, a bunch of document-sized baggies, and a sharpie pen.
2. Whenever you want to add a document, put it in a new baggy, write an autoincremented number on the front with the sharpie, and throw it in the box.  That’s it.
3. If you like you can keep an index of the documents on your personal wiki.  Keep a table with three columns: document number, date added, and a text field description of what’s in the baggie.

For me this method beats the crap out of the traditional filing system with the filing folders, little pieces of plastic, and perforated sheets of labels.

Here’s a sample list of things in my filing system, pasted from the descriptions in my index.

• Zeo / personal sleep handouts / paper / sleep tracking forms
• Lenovo x61 recovery DVDs
• Stock certificate
• Friend’s memorial service handout (this one for sentimental value)
• Heavily used pocket/folded maps of Stockholm and Hamburg. (again for sentimental value)
• 2009 Income Tax binder, federal and state
• Christmas cards, 2009
• Etc

Ok, that’s it for now.  Please share your own document organization tips in the comments!  Thanks!

Adam's Life Hacks Part One: Personal Wiki Time!

I would start with a story about how bad my memory is, except for the small fact that I can’t remember any such stories.  Suffice it to say that any significant other of mine can always pull a trump card over me by invoking the “You don’t remember when we XYZ?”  But now I'm venting.  : )

I use a mixture of methods to keep track of things in my life.

At any given time I have todo items in my email, a Christmas card list scribbled on a piece of paper blotted with chicken tikka marsala sauce, and some note in my head that it’s time to take the car in for an oil change.

And somehow that oil change never happens.  (With apologies to my whip.)

Long term personal storage was always tough, though.

Email isn’t good for long term storage because it’s cluttered with lots of other data, and it’s not editable.  Paper notes aren’t scalable enough.  And so on.

Enter the personal wiki.

I inaugurated my personal wiki two and a half years ago.  Since then I’ve created 480 pages and made 4,046 revisions which works out to over four edits per day.  I’ve found it to be incredibly useful.

Now lots of other people have written about personal wikis, so I’m going to just assume that you understand the basic idea, and talk about my tips on how to effectively use one.

1. Use MediaWiki.  At least as of 2008 it was by far the best.
2. Turn rich text editing off.  Use the text-only markup editing mode.
3. The only markup you need to know is: ===A section title===, [[Page Link]], [http://cnn.com HTTP Link], __NOTOC__ to disable a table of contents, start lines with a “* “ to get bullets, and use two blank lines to get a blank skip line.
4. Split your Main Page into two columns, like so,
   
   
   
   
   
5. Avoid hierarchy of pages.  Especially if you’re an engineer.  Your Main Page should be the only page that is just a bunch of links to other pages.
6. You must have the following pages:
   
   
   • People to see in different cities
   • Meeting Notes
   • Archive (where you keep links to pages that are now unused)
   • Logins (where you keep airline mileage account numbers, special passwords, etc)
     
     
7. The following pages are recommended but not required:
   
   
   • Startup Ideas
   • Potential Future Investors
   • Potential Future Hires
   • Grocery Shopping List
   • Mentors
   • Local Businesses
   • Useful Software
   • Places to travel to
   • People I met while traveling
     
     
8. Hosting: set up SSL and make your site https only.
9. Security: disable account creation, and disallow anonymous browsing.
10. Register a domain name that is dedicated to your wiki, that only you use.
11. Run the wiki on a fast server.  I started on shared hosting and found that my page load times were multiple seconds.  I now run my wiki on an old desktop machine, but an Amazon EC2 micro instance does just fine!

 

That is it for us today, I’m Adam Smith.  Thanks again for reading.  We’ll leave you with Sting, and a cut off his new album*.  Take it away.

* Rather than Sting, I’ll actually leave you with a censored list of pages I keep on my wiki, or at least the first few alphabetically.  Take it away.

 

• "Bookmarks"
• 10 Years of Software
• 2008 Q2 Review
• 2008 Q3 Planning
• 2008 Q4 Planning
• 2008 Review
• 2009 Q1 Planning
• 2009 Q2 Goals
• 2009 Q3 Goals
• 2009 Q4 Goals
• 2010 Birthday Invite List
• Academic papers read at the beginning of Xobni
• Acronis Easy Migrate says "Error: partition configuration changed" on Lenovo
• Acting There Will Be Blood
• Addresses to change when moving
• Apartment hunting 2008
• Apartment Party Post Mortem
• Apartment Recurring Tasks
• Apartment Setup
• Apartment Warming Party 2008
• Archive
• Art I Like
• Attending a Concert, How To
• Audiophile
• Beijing Trip Thoughts
• Better Focus
• Birthday Dinner Planning 2009
• Biz School
• Blackberry: Curve 8900 or Bold
• Blog posts to write
• Blog Todo
• Board Meeting, Oct 2008
• Books That Have Shaped How I Think, Tim O'Reilly
• Books to Read
• Budget Sept 2008
• Budget thru end of June
• Budgeting
• Burning Man
• Burning Man 2008
• Burning Man 2010
• Call with [XYZ], 29-apr-09

 

...okay you get the idea.

Please leave any tips of your own or other thoughts in the comments!

 

 

Random But Important, Part 1: Seeing Death

I used to think I would live forever.

When you're young, not only is it emotionally comfortable to ignore death, it's also helpful.  It makes you plan for the long term.  For example, you should get good grades, get a good education, and build good character habits because those things are going to be especially important if you're going to live forever.

This bias continues into adult life.  The infinite life abstraction is a good counterbalance to the cognitive bias of going for short term rewards.

But it breaks down and can lead you astray in a few places.

First, let's put this into perspective.  Here's a picture of your life.  Every square represents one month in the average American lifespan.

Seriously, think about where you are in checking off those boxes, and think about how quickly you're punching through them.  And then think about the things you still want to do in life.

It's very sobering.

How do we react to this?

If you're only going to remember one thing, remember to apply the "short life" perspective broadly, deliberately, and on a regular basis.

Broadly because mortality effects everything, deliberately because it's easy to forget in day-to-day life, and regularly because life unfolds over time.

Now, for the rest of this blog post I'm going to take the discussion in a direction that is mostly about startups and careers.  So be forewarned!

First up, make sure you're having fun.

Realize that life is a journey to be enjoyed, not a destination.

Thus, you've really got to do what you love.  It's okay to drudge through some shit for the short term, but make sure you're working in the right direction.

In particular, if you're working in an industry or company you don't believe in, try to tack towards a new direction.  Don't settle for boring or irrelevant.

Second, I believe that understanding death can lead you to the right kind of ambition.

Sure, be ambitious, because life is too short to be broadly risk averse.  We might as well be bold, and try to do something great.

But most young entrepreneurs don't need to be told to be bold.  Sometimes we have a problem with over-ambition, and I want to talk about that for a bit because sometimes it is fueled by a mistaken perception of immortality.

Someone who's overly ambitious wakes up on any given day, and forgets that they're going to die.  They're going to live forever AND kick ass!

And fuck it; if you're immortal you can reach perfection over time AND eliminate the effects of luck over time.  You're going to end up a mogul!  How can you not?!?

...except, you're not.  You're going to die.  Ouch.

You only get one childhood, for example.  No do-overs.

And your window to make a huge impact is very, very small.  99.99% of people don't hit that window.  Really, it's the exception.

So be humbled, and check your youthful infinite ambition at the door.  There's actually very little time for bullshit.

For example, if you're going to get anywhere, you need a great team.  You can't do it on your own, mortal Molly.

You'll need to do everything necessary to make your team successful.  That includes making sure the needs of your team members are met, even when they're in conflict with your own ego.

It's telling that Steve Jobs, the ultimate icon of a badass, brings his team members onto the stage at his keynotes to show everyone their great work.  And he negotiated for his executive team at Apple to get fresh option grants in 2001 before he said anything about his own compensation.

Well, it was a tough situation, you know.  It wasn't so much about the money, because a very small percentage of my net worth is from Apple.  But everybody likes to be recognized by their peers, and the closest that I've got, or any CEO has, is their Board of Directors.  And as we've seen in the discussions of the past hour, I spent a lot of time trying to take care of people at Apple and to, you know, surprise and delight them with what a career at Apple could be -- could mean to them and their families.  And I felt that the board wasn't really doing the same with me.

(..)  So I wanted them to do something and so we talked about it.

- Steve Jobs deposition, page 89

So, teamwork matters at least because you can't live forever.  We basically derived from first principles that you can't do it all on your own, and explained why people with overgrown egos might approach company building from the wrong angle.

And, finally, it's okay if you fail.

It's even okay if you don't go for a huge impact.

Most people are net contributors to society, economically and spiritually, and that takes us all a very long way.

You've just got to do your own thing, whatever that is, and push things in a positive direction.  Life is too short to be driven by extrinsic motivation.

...ok, that corniness high note is my cue to wind this down for now.  Can you tell that I've been thinking about company building and leadership lately?  : )

 

To conclude for today, you're going to die so try to be rational about it.  It's hard to remember on a daily basis that you're going to die, but try to regularly ask yourself if you should be doing anything differently given that life is short.

Put a monthly recurring appointment on your calendar if you have to.

And remember to apply it broadly.  Forgive, for example.  Be open and honest.  This one has wide implications.

 

More on this topic:

• Regrets of the Dying
• Regret Minimization Framework
• Steve Jobs Graduation Speech, especially the third story

 

If you have any other thoughts or links on this topic, please share them in the comments!  Thanks!

I Love MIT!

God MIT is a great place.

Last week I got to speak at a new MIT class, The Founder's Journey.  There were 100+ students.  Pizza was had.  There was a buzz of energy and ideas.

We had a lot of fun, and I'm excited that my friend Drew Houston will be speaking there tonight.

After the class last week some students invited me back to their dorm, Burton Conner.  Beers were had.  And a jolly time ensued.

 

 

Included in this picture but not immediately obvious: a homebrew video game arcade with CRT monitor, and a huge tank with live turtles.

Hackers are alive!

Reason #2 why I love this place: earlier this year at MIT I learned why in about 40 years we'll all be flying in vertical takeoff airplanes.  : )

Crazy Idea Sunday

"I tell you something that is not right out apparent, but it's getting a huge amount of focus within the company - and it is apparel. We are having a lot of sales growth in apparel.  (...)  Historically clothing has always been a number one mail order category even pre-internet. And we make it really easy for people to try stuff on - by making it easy to return."

Jeff Bezos, Charlie Rose Interview, July 2010

Ok, time for crazy idea Sundays!  This will probably never work, but it's fun.

People are going to be buying more and more apparel online.  According to the Bezos quote above, Amazon is working on this category.

Call me crazy, but here's what you do.

Take the millimeter wave scanners the DHS is trying to get into airport security lines, and set up a couple in each major city.  Get people to go by one of these scanners and get their body 3D scanned.

Then you get a 3D representation of everything in your store, and some idea of how the clothing behaves under various conditions (lighting, stretching, wind, etc).

Then when someone wants to shop you perform some solid-body physics simulations on your server, and render images and video of what the person would look like in any given set of clothes.

...

Well, at least you'd have more success than the DHS with those things...

The More Important Reason Electronic Books Will Be Great

Call me an optimist, but I think we're about to see a change in how books are written and consumed.

There's a ton of data exhaust coming out of electronic book readers describing, most importantly, how much of each book actually gets read.

Imagine looking at a book on Amazon and seeing "We predict that you personally will read 5 pages of this book."  Or, on the flip side, "You will read 98% of this book."

Holy shit!  So much better than average star ratings.

But wait!  It gets better!

Publishers are finally going to be incentivized to make books as short as optimal.  I don't know much about the publishing industry, but from what Dharmesh told me nonfiction books are too long today because people actually buy on length.  That might finally change!

Thoughts After Y Combinator Demo Day

There's lots of motion in the ocean in the early stage investing world.  TechCrunch has a write-up on the companies that presented at Y Combinator's demo day yesterday.  I had several of my own thoughts.

Quality

• The average company quality is up.  A much greater fraction of these companies will be singles or doubles compared to years prior.  I'm not sure if there will be more % homeruns or not.
• There are also more companies.  Xobni's class of 2006 had 15 companies; yesterday's graduating class had 36!
• So there's more early stage investing opportunity at YC than before.

The Index Fund Strategy

• Ron Conway and Keith Rabois invested in a number of these new YC startups.
• Some folks call this strategy the 'index fund' approach.  Let's define it as investing in more than five out of the thirty six companies.
• The 'index fund' approach might produce better returns (and/or risk-adjusted returns [1]) than cherry picking the best one or two companies from each Y Combinator batch.  I'm not sure.
• The index funds have three main advantages.  First, if you assume everyone's throwing darts then they are more likely to invest in the homeruns.  Second, they can collect returns from the longer tail of singles and doubles without breaking a sweat.  Third, they can make investment decisions faster.
• Interestingly, VC's still try to cherry pick.
• As are some of the new super angels.
• Anyone in the comments want to take a hack at the index fund versus cherry picking strategies, e.g. in environments with different levels of froth?

The World is Flat

• VentureHack's Angel List and YC's Demo Day have made the fundraising landscape more flat.  It's easier to see a wide variety of deals as an angel without doing heavy networking.
• This creates more competition.
• In particular is removes a competitive advantage for heavily networked angels like Ron Conway.
• Networks and experience still matter a shit ton for providing value to portfolio companies, though.
• For example, a new investor from Hollywood could come in and start making lots of investments very quickly.
• But when shit hits the fan or you need backchannels to an acquiring company the movie star won't be able to help.  (Unless the acquirer is Roc-A-Wear and your investor is Jay-Z, in which case I'm pretty jealous.)

Investor Credentials

• Looking at a VC's investment list is very useful.  Did they get into the hot deals?  Were their companies successful?  What do their founders say when you call for a reference check?
• Angels advertise their investment histories as well, but the informational value of those lists will go down as the world gets more and more flat, AND as the index fund strategy becomes more popular.
• Being an angel investor in Google back in the day meant you were connected.  It meant you were friends with the founders.  It meant you know some of the behind-the-scenes details of the company.
• But angels are investing in more companies, and companies are raising money from more investors.  So again the informational value goes down.
• What's the new credential?
• Probably references.  Founders of an angel's companies can tell you the real stories behind how an angel hurt, noop'ed, or helped their company.
• Do we need a TheFunded for angel investors?

Multiple Valuations: Early Bird Gets The Worm

• Another factor: different valuations for different angels.  As Paul Graham recently pointed out, a startup can issue convertible notes at different caps to different investors.
• This degree of freedom could be used in a number of ways.
• Some companies are trying to use this to give different valuations to different investors as a function of the order that they invest.
• The idea is that the first investor to commit is taking the most risk, so they should be rewarded for that.
• This can only go so far.
• If a company needs a minimum of $X to get to the next step, once a company has raised $X a valuation premium doesn't make sense.
• $X for a YC startup is probably around $200k.
• I'll stop there on this topic.  There are lots of dynamics at play, and it will vary company by company, but in summary I don't think early bird valuation discounts will become super prevalent.

Multiple Valuations: Value Add

• But as the angel investing world gets more flat the average investor will become more vanilla, and investors like Ron Conway will become more differentiated by their savvy, connections, and in some cases the time they can devote to helping the company.
• I predict that it will become more common to reward value added investors with valuation discounts.
• ESPECIALLY if these value add investors commit earlier.
• And often the big value add investors will commit earlier because they understand the space and get really excited about it.
• For example, one YC company yesterday had already raised money from a big time executive in their industry who can help them with intros/advising/etc.  Those situations can call for valuation boosts.
• Valuation boosts, while rare, were previously done by issuing extra common stock (sometimes specifically called "advisor shares") to the value-add investor.  But that approach was a little too heavyweight if the angel wasn't going to be heavily involved.
• So I think valuation cap differences will become more common for compensating value added investors, especially when they are the first money in.
• Secondary effect 1: non-value-add investors will be paying more for their equity, at least in theory, since entrepreneurs can now price discriminate.
• Secondary effect 2: more angel investors will try to be value add.  (?)  Especially super angels who have more resources and more companies to amortize fixed cost value adds across.

...good times.  : )

[1] thanks to Will Stockwell for reminding me that, of course, maximizing return/risk ratio is more important than maximizing returns.  More on this topic in the context of VC over at Fred Wilson's MBA Mondays.

Six Months Later: Seven Major Websites that Send Passwords Unprotected

It's been six months since I first wrote that 7 out of the 36 most popular websites sent login passwords in the clear, without SSL encryption.  This basic weakness means your password can be read by anyone who can see your Internet traffic, for example anyone using the same wireless access point as you.

I mentioned that I wanted to bring those seven sites around to having secure logins.  This is our six month progress report.

I also want to expand our ambitions to tackling the next 1,000 most popular sites that are insecure.  We're going to need a new strategy to do that.  More on that later, after the results..

Drum roll please...

Three out of the seven top websites added secure logins.  As far as I can tell, in each case the change was due to this blog's original post and the follow up efforts of myself and our readership.

That – ladies and gentlemen – is awesome!!!!!

Bit.ly stats aren't working for me right now but I know the blog post was retweeted 200~300 times in many languages.  This all happened when everyone was talking about China's internet attacks, so frankly I don't expect to get as much attention on this follow up post, but I do think we can nevertheless maintain the same pace of progress in the next six months.

More on the next six months below.  For now I want to highlight the status of each of the seven websites, and offer thanks where thanks is due!!

The following websites now have secure login forms:

• Photobucket (thanks to Sachin Rekhi, Luke Swanson, and Normal Liang!)
• GameSpot (thanks to AirGuitarist87 and Jody Robinson!)
• Taobao.com (I'm not sure who to thank here)

The following sites still have insecure login forms:

• Slide
• hi5
• Tudou

Wikipedia is also technically in the "still unsecure" list, but I decided to put less emphasis on them since (a) they provide a prominent link to a secure login form next to the login box, and (b) most of their users don't use logins.

I was able to contact someone at Slide, hi5, and Tudou but I wasn't able to get direct results from those conversations. (See update below.)

Broader goals, and a new approach

There are many other websites in the top 1,000 that send passwords in the clear.  Some of these include digg, reddit, tumblr, and plentyoffish, but there are many others.

My original ambitions were only for the top 36 ranked sites, but it's worth going after the more ambitious goal of securing the top thousand if we can do it, and I think we can.

My emailing these company directly is not a scalable approach here.

My favorite new approach is to propose a patch to Firefox that will gently tell the user after their password is sent in the clear, and suggest they contact the website to ask that they secure their login form.

I created a post on the Firefox dev mailing list and a good discussion ensued around the usability and technical issues.  Overall support was moderate to strong.

...changing the world is possible!...

I must admit, though, that I need your help.  I'm not familiar with the Firefox code stack and don't have the immediate resources to change that.

Do you know someone who might be able to help, or do you have any ideas for how to get this done?

Please let me know if you do.  I have ideas of my own but welcome all suggestions you may have.  You can contact me on Twitter or via email.

More in another six months!  Thanks!

(Update: an engineering from hi5 has told me that they are working on it for their site!)

The Upcoming Generation of Lossless Data Compression, Part 2

My last post about the future of lossless compression algorithms was about cross-file content-aware dedupe.  (Hint: this is different from what most people think of as just 'dedupe'.)

Part 2 is about lossless compression algorithms and systems that support incremental addition and removal operations.

For example, in NTFS, probably the most popular file system in the world, if I turn compression on and store the same file twice each file will be compressed and stored individually, leaving additional compression opportunities on the table.

Another example: MySQL doesn't support compression for BLOBs of data.  The first step towards this would be compressing each BLOB field individually.  The second step would be to look for redundancies across BLOB objects.

PostgreSQL, another popular database package, does the first part but not the second.

In many cases this second step will be hugely important.  For example, FriendFeed circa 2009 stored lots of 300 byte JSON blobs.  In this case the difference between compressing two million 300-byte blocks of text individually versus together could be huge.  (More background on FriendFeed's use of MySQL here.)

Of course this feature is fairly obvious when you're looking at something like a file system or database where adds and deletes are natural, but it's hard to pull off in practice due to the risk of inconsistencies.

(This is precisely the reason it should be done at the database engine layer, rather than the application layer.)

But just because it's hard doesn't mean it won't be pervasive in MySQL deployments in 10 years or so.

Final thought -- I'm sure there are technologies out there that do this already.  The real shift/disruption will come when it becomes part of a non-proprietary standard, just as with 7zip/LZMA, and/or integrated into popular database engines.

The Upcoming Generation of Lossless Data Compression, Part 1

First, yes – there is such a thing as next gen lossless data compression.

Here’s one for starters: cross-file content-aware dedup for images and video.

Take a JPEG image, compress it with your favorite lossless compression algorithm, say BZip2 or LZMA, and you’ll get a larger file.  Of course that makes sense, as JPEG itself is written to be compressed out of the box.

Now take a group of, say, 100 JPEGs, and compress them into a single archive.  You’ll get no real compression.  Even though there are likely redundancies among those photos, JPEG isn't designed to handle cross file compression, and LZMA can't handle it because the redundancies aren't obvious by looking only at the binary patterns.

But do that with a compression algorithm from the future, and you’ll get a much smaller file.  These algorithms will know how to do content-aware cross-file compression.

What would these algorithms look like?

In fact we already have algorithms that know how to de-dup among a set of pictures.  They’re called video compression algos.  Try taking those same 100 JPEGs, put them into a movie file format, and run your favorite video compression algorithm over them with the right parameters and you’ll get your smaller compressed archive.  Of course this is just a hacked approach, but you get the picture!

There are also various tricks you could do on text, audio, etc.

This technology will become increasingly important as more and more data gets centralized into the cloud.

Someone from the commercial side is already working on this.  After thinking about this idea for several hours and getting really excited, I came across a company called Ocarina (chalk talk here) who is commercializing this approach.

That doesn’t mean this technology should be limited to big companies with big budgets, though.  I’m hoping someone from academia or the open source community will eventually pick this up and run with it!

Part 2 to come soon!  (Edit: link to part 2)  In the mean time let’s hear it if you have any further thoughts or ideas!

(There was a good discussion of this blog post on Hacker News here.)

From Zero to a Million Users - Dropbox and Xobni lessons learned

I just finished a talk at Web 2.0 Expo titled From Zero to a Million Users with Drew Houston, the founder of Dropbox.  It's about what we wish we had known years ago when it comes to growing your user base from 1 to 1M users.  We had a lot of fun with it.  The slides are below!

From Zero to a Million Users - Dropbox and Xobni lessons learned

View more presentations from Adam Smith.

Amazon S3’s Pricing Model is Arbitragable, and the Future of Cloud Storage

Here’s what you do.

Amazon S3 charges money for each gigabyte that you store.  You go to their top 50 customers, and tell them: we’ll dedup your data with the other top customers, resulting in overall cost savings.  We’ll split those cost savings with you 50/50.

You're basically stealing the economies of scale of redundancy elimination from Amazon.

You could also get sophisticated with delta encoding blocks of data that aren’t duplicates but are close.  Here’s a paper that goes detail on redundancy elimination.

You’d have to get people to trust you to not lose their data.  Maybe you use best practices, hire cperciva to write it, and get an insurance company to provide a $100 M policy against data loss that’s your fault – but it’d still be hard to pull off.

This line of reasoning raises questions about the future of cloud storage.  Mostly: does broad scale redundancy elimination matter?

If it does matter, we will see economies of scale in cloud storage.  Does that mean there will be one or only a few major providers, or will redundancy elimination be provided as a layer on top that’s operated by someone else?  Will cloud storage providers be forced to change their pricing models?

These are the questions that make our field so exciting!

Just to sweeten the pot, Data Domain, a data deduplication company, got bought for $2.4B a couple years ago, though they were certainly approaching a different market.

What do you think: will large scale redundancy elimination matter?

The Market Opportunity to Undercut Sonos

New technologies often start out serving the high end of the market where quality is more important than price, and over time a new disruptor comes in to serve the lower end of the market.  This is a classic part of the innovator’s dilemma.

It’s also pretty universally understood that geeks love their Sonos sound systems.  These boxes of magic synchronize music playback across speakers spread around your home, all wirelessly.  For example, right now I’m playing music from my PC into a Sonos line-in that gets piped into my living room speakers.  If I walk upstairs to do laundry I can pull out my iphone and use the remote control app to route that music into my upstairs speakers as well.  Playback is synchronized.

A low end Sonos system costs $800.

They also have all kinds of fancy features like integration/partnerships with Rhapsody, Napster, Audible, Sirius Radio, etc etc.  Their boxes have optical audio ports, ethernet ports, etc.  None of these things will matter on the low end.

So what, exactly, would the low end solution look like?  Easy: small wall-warts that have a wifi chip, an audio card (D2A and A2D chips), and a microcontroller/DSP to make it all work.  On the outside it would have two audio jacks: one for audio in and one for audio out.

(You might also sell a set of speakers with your device built in, since the less hacker-ish part of the market will want a total solution.)

Now, I have zero experience with hardware but I’d guess you could get the bill of materials down to $40, without speakers.  If you are super scrappy you should be able to bring the initial product to market for $750k.  But I could be totally off base.

The goal would be a cost point that's ten times cheaper.

Here’s the coolest part.  Using Sonos’ audio streaming today requires two Sonos boxes – one as the audio source and the other as the sink.  A standalone computer can’t be a source; if you want to play music from your laptop and move around you’re hosed.  (There is a way to do this, but you have to set up and use their player software, which is not the right solution.)

The low end disruptor would need this “virtual sound card” feature, for both PC and Mac OS.  The lowest common denominator would be having one of your devices plugged into your stereo system and you can stream music to it from your laptop.  And people can upgrade over time by just buying more of your cheap devices.

The only other meaningful player in this space is Apple’s AirPort Express, which can stream itunes to an audio jack.  I don’t believe you can play from any audio source on the computer, and I don’t believe you can stream to multiple AirPorts at the same time.

I’ve had this idea on my ideas spreadsheet for years, but Sonos just announced a major financing so I’m concerned that they might be eyeing this strategy, and frankly I need someone to scratch this itch for me.

So here’s the deal.  Let me know if you’re interested.  There are a couple of other interesting product and marketing angles that we could jam on, and I might want to put some money in.