Chapter 1: Poindexter

Smoke and mirrors

In honor of the 20th anniversary of the ad exchange, I’m posting the first few chapters of my unfinished book on the history of programmatic advertising. Here’s chapter 1. All of this is my own recollection and I am sure I am misremembering things - please forgive my errors!

The Interview

It was an interview, so I wore a suit. When Matt Philips, who I had met through a Princeton careers email list, walked into the room, I knew it was a mistake. He was slim, in his late twenties, and dressed like a cool dot-com executive.

Matt spent a few minutes explaining the genesis of Right Media. Over the past few years, he had been a product manager at DoubleClick, working on the company’s flagship ad serving product. I knew a little about DoubleClick. They were one of the hottest dot-com companies in New York, providing the software that enabled web sites to track and deliver ads.

Matt explained that DoubleClick also ran an ad network, aggregating ad space across these web sites and selling it to advertisers. Mike Walrath had been a top salesperson at the DoubleClick ad network, specializing in performance advertising: getting people to click on ads and buy stuff. He and Matt had started Right Media to build a performance ad network that would outperform DoubleClick - and now they needed a tech guy.

Matt wrapped up his explanation: “So, we realized that the DoubleClick ad server didn’t work very well for ad networks. We looked around for a company that could do performance optimization, and Poindexter Systems seemed like the most advanced company in the industry. We sublet a little office space from them, and they’re going to build us the software. In the interim, we’re running the business without any tech. What we want to hire someone to do is basically build us a web site and maybe help automate some of the manual work we’re doing.”

I launched into my spiel. “Over the past 18 months, I built an enterprise software platform that we have been selling into large companies as an add-on to their financial systems. Before that, I was a consultant building predictive personalization systems. I have a computer science degree from Princeton, as you know, and my research focus was large-scale distributed systems.”

Matt looked at me, raised an eyebrow, and gave a slightly embarrassed laugh. “Uhh… that’s great… but we probably don’t really need all that.” He thought for a moment. “You know, you should really talk to the Poindexter people. They are building the really complex tech, and I bet they could use some help. Let me go get their CTO.” Matt stepped out of the room, and a few minutes later, a bearded man entered in his stead.

“I’m Steve Jacobs, the CTO of Poindexter. Matt tells me you’re a software engineer looking for work, with personalization experience?”

I explained my background again, and after a bit of back and forth, Steve offered to bring me on as a consultant. When I returned to New Jersey that afternoon, I realized I barely knew anything about Poindexter. I had been so confused by the bait-and-switch interview that I had failed to ask basic questions, like who was the CEO or what the company was named for.

Personalized advertising 101

The next morning I got back on the PATH train to 23rd street, this time dressed more casually. I walked into the Poindexter office and Steve gave me the quick tour. Joe Zawadzki, the founder, sat in the back corner. He was tall and intense, with brown hair just on the verge between styled and mussed, long and short. He wore a tweed jacket with leather elbow patches and thick-rimmed glasses. He clasped my hand warmly and welcomed me on board. As I walked in a circle around the desks, I met the team. Steven Golus, sales. Ted Shergalis, COO. Then I met the engineers I would be working with, Alan and Melanie. They were vibrant, perhaps thrilled to have someone to help carry the load. We went into the conference room and they began to explain what the company’s software actually did.

The basic premise behind Poindexter was that digital advertising should be optimized so that each consumer sees the ad that she is most likely to respond to, unlike television or print where every consumer sees the same ad. To achieve this, they put a snippet of code on the web site that would run some logic to show the optimal ad to each visitor. The Poindexter technology had a few components: the real-time execution engine that showed the ads; a data collection system that tracked ads and clicks; a user interface that customers could use to upload ads and see reports; and the secret sauce: the Poindexter Optimization Engine (POE) that did the optimization to figure out which ad to show to each visitor.

I began to ask questions about each component of the system. The real-time execution engine was built on top of Akamai, the content delivery network, using a specialized language that could insert dynamic content. This was cloud computing, in effect, just ten years too early. The idea was that Akamai had servers all around the world, and could handle the technical overhead of real-time content delivery. The downside of being ten years early was that Akamai’s service was extremely expensive and quite limited, forcing most of the complexity into POE. I asked how the data got from the Akamai cloud into reporting. Melanie explained that the logs were downloaded on a daily basis, processed, and then aggregated into a database for reporting.

Next, they showed me the user interface. Users could create an advertising campaign, decide which geographies and times of day the campaign would run, and then associate the actual ad itself (called a “creative” in advertising parlance). The final step was a massive screen called the Grid that listed all of the creatives on one axis and every placement (the actual spots on the websites where the ads could display, like “New York Times right rail”) on the other. In the middle of the Grid were hundreds or thousands of checkboxes that determined whether each creative could run on each placement. It was so massive that it almost crashed the browser.

As a software architect, my immediate reaction to the system was mixed. On one hand, I knew nothing about this industry, and from everything I had heard this was the state of the art. On the other hand, this felt like a system that was duct taped together more than designed. I was a bit concerned, and at the same time, saw clearly why they needed my help. I began to throw out ideas to improve the system, starting with fixing the user interface so it would actually function. Alan and Melanie agreed that this was a good place for me to start, and I sat down at my desk to work on a proposal for a cleaner, better-architected UI.

Over the next few days, as I worked to redesign the UI, I began to get a feel for Poindexter. Each day at lunch, Alan, Melanie, and I would go out in search of food. The Madison Square Park area was still an extension of the Garment district, and finding a quality lunch was a challenge. Eisenberg’s sandwich shop was always packed, as it probably had been for decades. There was a Chipotle, perhaps the first in Manhattan, a 15 minute walk over to 7th avenue, so we usually went to Uncle Moe’s on 19th, or down 5th avenue to a decent pizza place. There was one Starbucks within walking distance, on 6th avenue, and the delis just downstairs did a hopping business with their hot buffets.

Each day, a low-flying airplane would zoom across the sky and pedestrians would reflexively pause in their tracks to watch, calculating whether it was on track for a building. This was Postlapsarian Manhattan, that time just after the fall of the Twin Towers and the dot-com collapse, when airplanes and entrepreneurs were echoes of disaster.

As I walked to lunch with my colleagues, I learned more about Poindexter. Founded by two Harvard alumni, Joe and Ted, the company rose on the optimism of the dot-com boom. When the market collapsed, they were just able to keep the lights on, skimping on office furniture and subletting part of the office to stay alive. Rumblings of missed payroll were quickly squelched. I had joined a group of survivors, adrift, scavenging for lunch in the wasteland of Silicon Alley.

A few weeks into my time at Poindexter, a preppy 40-year-old man walked into the office looking like a hungover Robert Redford. He made jovial comments as he walked along the desks, then stopped in front of me. “You must be BOK!” he exclaimed. I was taken aback. These were my initials, I supposed, but what the hell? “I’m Rich Person!” As I stared at him, trying to understand why he was calling himself a rich person, I remembered an article I had read about Poindexter hiring a new CEO from an ad agency. Wasn’t his name… no way. He was a walking caricature of a New England executive, and that was his name? “You know, I drive a Porsche Boxster. BOKster? Get it?” And then he walked away.

I leaned over to Melanie. “That is the CEO of this company?” She nodded and shrugged. “Why haven’t I seen him before? Does he even work here?”

She whispered back, “He lives in Boston or something, and commutes to work. The board wanted to replace Joe with a more experienced executive.”

The dirty secret behind POE

A few days later, Steve Jacobs walked by my desk. It was spring, and the sunlight began to pour through the front window of the office. I was beginning to make progress on my user interface redesign. Steve told me that our technology advisory board was in town, and asked me to join a whiteboarding session with them. The two of us went into the conference room. The advisory board, three computer science professors, introduced themselves.

The conversation began with a discussion of cutting edge optimization methods. The professor from Dartmouth laid out the handful of approaches that were most common: logistic regression, decision trees, and, … “wait! What was that? Decision trees?” exclaimed Steve. We all stared at him.

The professor, surprised, went to the white board and began to draw a decision tree. At the top of the board, he drew a circle and wrote “time of day” inside. Then he drew three arrows down from the circle: morning, afternoon, and evening. He explained that a decision tree is like a flow chart, where the algorithm can only follow one path out of each circle. Below morning, he drew a circle and wrote “State”, then drew a number of arrows below it: Alabama, Alaska, Arizona. Beneath each of these arrows he drew a circle and wrote “Ad 1”, “Ad 2”, “Ad 3”. The idea, as I figured out, was that you could follow the chart, making a decision at each step, until you figured out which ad to show. Because each step branched off into multiple choices, the whiteboard looked like an upside-down tree. Thus, decision tree.

I found this interesting. Steve, next to me, looked like he had seen a ghost. I gave him a querulous look, and he said, “This is how POE works!” I was extremely confused. I almost blurted out a question. How could Steve Jacobs, the CTO of the company, not know how the company’s secret algorithm worked?

I managed to hold my tongue through the rest of the meeting, but as soon as the professors left, I cornered Steve. “You don’t know how POE works? What is going on?”

He pulled me into the conference room. “OK, here’s the deal. One of the founders wrote POE, but he left and didn’t leave any documentation behind. I’ve been trying to figure out how it works.”

My mind was boggled. Every press release, every article, the very core of this company was this incredible optimization system. “So how do you operate POE if you can’t maintain the code?”

Steve sheepishly explained. “Well, I run the POE software on each campaign after about a week of data has come in. It outputs a decision tree in XML. Then I edit the tree by hand to make sure it meets all of the customer’s requirements.”

I felt like I had been punched in the face. This whole company was a farce. There was no secret sauce, no sophisticated technology. I felt like Toto in the Wizard of Oz. Not only was there a man behind the curtain, it wasn’t even my story. As I poked around on my computer the rest of the day, I tried to decide what to do. Should I tell Matt that he was betting Right Media on a sham of a platform? Was that disloyal to my employer? Was it unethical not to tell him? I asked Steve if he had told the Right Media team that the software wasn’t going to work the way they expected. He shrugged and said that it wasn’t his call.

I’m out

The next day, I grabbed Matt and asked him if he wanted to take a walk. As soon as we were clear of the building, I told him what I had found out. “Matt, they don’t have the software they promised you. I don’t even think they’ve started working on it.“

Matt was shocked. As we walked around the neighborhood, he probed for more information. As I explained to him that the entire process was manual, he looked deflated. “I understand what you’re saying, but what can we do? We don’t have the funds to build our own ad server, and there aren’t really any other options on the market.”

I looked at him. “I could build it.” Matt raised an eyebrow and frowned. “No, seriously. Their technology is terribly broken. I know they’ve put a few years into it, but if they can’t even maintain their core algorithm, it’s never going to do what you want. I’ve built an ad server before. I’ve built enterprise software. I can do it for a lot less money than you’re going to pay them.”

Matt pondered this. I could tell he was looking at me, all of 25 years old, and trying to get his head around the idea that he should trust me instead of Poindexter. After a moment, he shook his head as if clearing it from a dream. “Let me ask Poindexter some questions. They keep telling us the software is almost done. Maybe you just haven’t seen everything they’re working on.”

The next couple of weeks were painful. I continued working on my project, but my heart wasn’t in it. I started thinking about looking for a new job, or maybe starting a company. Rich Person walked around handing out invites to a company party at a restaurant called Tao. I think the idea was to help employees let go of some of the recent issues, like missed payrolls. The invite read: “That was zen, this is Tao!”

The morning after the forgettable, sad party, I bumped into Matt in the elevator and felt compelled to tell him about it. He laughed, then leaned over and whispered, “can you meet Mike and I in Madison Square Park at 2 PM?”

I looked around the empty elevator. “Um, sure.” He leaned back against the elevator wall and we tried to act non-conspiratorial as the elevator crept up to our floor.

At 2 PM, I crossed over to Madison Square Park and saw Mike and Matt standing by a statue near the dog park. I had seen Mike going in and out of the office, but we had barely spoken. He was six feet tall, with dark curly hair, athletic. He wore khakis and a button down shirt over loafers, and seemed slightly uncomfortable waiting for me as I walked over. I felt a slight shiver of anticipation as I shook his hand. “Matt tells me you can build us an optimizing ad server.” I nodded. “How much would it cost, and how long would it take?”

I tried to figure out a reasonable answer on the spot. “I think if I had one engineer to help me, it would take about three months. We would need servers and datacenter space to operate it. If you can give some sense of the volume that you’d want to run, I can get some numbers together.”

Matt and Mike looked at each other. I tried to guess what the look meant. Was my estimate too high? Matt said, “Um… that seems really fast. Are you sure it won’t take longer?” I explained my reasoning, walking through the key components of the system. I could sense their skepticism. I knew I could build what they needed, both from my own experience and from the few months at Poindexter working on a similar system. I just didn’t know how to convince them that they should bet their business on me.

Mike looked into my eyes with an intense stare. “What do you get out of this?”

I thought it for a few seconds. What I wanted was to solve an interesting, large-scale problem with technology. I had done no diligence on Right Media. I didn’t know Mike at all. I liked Matt from our few chats, but we weren’t close. I could go get a job as a software engineer or architect, and get paid good money. What I really wanted was enough money to be able to pay a mortgage on an apartment, and to work at a startup. Matt had shared that they were making some money running their ad network, and had a few hundred thousand in angel funding, so I knew they could afford to pay something. “If you can pay me $90,000 as a salary, I’ll build this for you.” I knew this was far below market value for a talented engineer. I wanted equity, or at least the idea of shared upside, but I didn’t like the idea of asking for it. I wanted to earn it. I wanted to prove myself. I wanted to matter.

We shook hands, there in Madison Square Park. It was time to get our hands on Pandora’s Box.