Can Google Stay on Top of the Web? - BusinessWeek

Image via Wikipedia

As Bing, Facebook, Twitter, and less well-known upstarts nip at its heels, Google has hundreds of wizards racing to come up with smarter answers

By Robert D. Hof

High on a wall of the lobby at Google's (GOOG) sprawling headquarters in Mountain View, Calif., a projector displays a live sampling of the 2.5 billion searches made on Google every day. One after another, every second, they appear and just as quickly scroll out of sight: "Route 81 closed," "cushing's disease and canine diabetes," "weather." It's a graphic reminder of how many people, some 720 million a month worldwide, rely on the search giant for links to information, entertainment, products, and just about everything else on their minds.

Yet upstairs from the lobby on this bright September morning, two dozen Google search engineers and executives are gathered around a long conference table, not to celebrate their success but to wrestle with their failures. Headed by Udi Manber, one of nine Google vice-presidents for engineering, these are the leaders of a cadre of engineers and scientists known as the search quality group. They are the masters of the mysterious mathematical wizardry that has made Google one of the most powerful companies in the world. And every week, in a quixotic quest to provide the perfect answers, they meet to grill each other on how to improve Google's search results.

Better than anyone, these folks know that while Google may outperform other search engines, it still spits out plenty of clunkers—irrelevant sites or even occasionally no sites at all for a particular query. They also know that every disappointing result means someone is less likely to click on ads—the source of nearly all of Google's $22 billion in revenues last year—and more likely to try another search engine. "If it turns out that somebody offers a better service than we do," Manber, a former academic and executive at Yahoo! (YHOO) and Amazon.com (AMZN), says with characteristic understatement, "that's a concern."

Today more than ever, Manber and the brainiacs in the search quality group can't afford to falter. Google's competition has recently gone from pitiful to plentiful: Microsoft's (MSFT) new Bing search engine picked up 1.5 percentage points of market share in August to hit 9.5%, according to market researcher Hitwise, while Google's share fell from 71.4% to 70.2%. Bing's gain is partly thanks to a $100 million marketing blitz complete with television ads knocking Google every which way but in name. Microsoft's pending deal for Bing to become Yahoo's underlying search engine, creating a combined entity with 27% market share, could produce Google's first sizable competitor in years.

In addition, there are new upstarts, such as Twitter, Facebook, and Wolfram Alpha, a "knowledge engine" that attempts to answer factual queries in a more organized, comprehensive way. These and other companies are offering search services in such specialized areas as breaking news, updates from friends, and scientific research. Twitter, for instance, has become some news junkies' go-to site for finding out about plane crashes and other news that Google's computers haven't yet provided links to. "Google's very good at searching content as if it's out of a library," says Kimbal Musk, chief executive of OneRiot, a search startup for real-time posts and news from Twitter, Digg, and other social sites. "Twitter let people know another kind of search is possible."

ALTERNATIVES TO SEARCH?

Of course, countless would-be Google killers have fallen on their faces over the years. But longer term, Twitter, Facebook, and related services may pose a more fundamental threat to Google: a new center of the Internet universe outside of search. Twitter, now with 55 million monthly visitors, and Facebook, with 300 million, hint at an emerging Web in which people don't merely read or watch material but communicate, collaborate with colleagues, and otherwise get things done using online services. "Today the Web is not just a collection of pages, it's a collection of applications," says Anand Rajaraman, CEO of Kosmix, a guide to specific Web topics such as health.

As the Web evolves, there's no guarantee that search, or Google, will remain at its center. Not so long ago, portals such as AOL and Yahoo dominated the Internet as most people's first stop online. They lost ground as search improved and helped guide people quickly to sites they were looking for. Now Twitter and others are becoming significant drivers of people's attention to Web sites—Google's raison d'être. The New York Times (NYT) recently said that some 10% of NYTimes.com's traffic was sent by Twitter. Some marketers sense in the rapid rise of Twitter and Facebook a new way other than search for people to find information, connect with friends, and get recommendations and ads for products to buy. Twitter's ability to raise a round of venture money that gives the startup, which has no meaningful revenue yet, a valuation of $1 billion reflects the potential some people see.

Google clearly understands this emerging new Web, given its investments in such services as the video sharing site YouTube, online applications, and even an operating system. But the company hasn't been able to generate significant revenue from anything besides search-related ads, raising doubts about how long it can remain the key leader of the Internet economy.

Meanwhile, Google's very success and size are starting to work against it. In the past year the company has been the target of three U.S. antitrust inquiries and one in Italy. Most recently the Justice Dept. on Sept. 18 said Google's controversial settlement with authors and publishers, which would have allowed it to scan and sell certain books, must be changed to avoid breaking antitrust laws. Even Google's own paying customers—advertisers and ad agencies—say they're eager for alternatives to blunt Google's power. Says Roger Barnette, president of search marketing firm SearchIgnite: "People want a No. 2 that has heft and scale."

The challenges come at a pivotal time for Google. The company, whose founders Larry Page and Sergey Brin have thumbed their noses at conventional corporate thinking since Google's founding in 1998, is going through a trying transition to maturity. Its year-over-year revenue growth skidded to 3% in the second quarter from 31% in 2008, prompting several small rounds of layoffs. Analysts chiefly blame the economy and expect search-ad growth to return quickly as the recession eases—one reason Google's stock has more than doubled from its 52-week low last November to nearly 500 today. Yet some current and former Googlers say the company's size has made it slower to act and thus less attractive to the entrepreneurial folk who flocked there in its early days. Google has seen a number of high-profile departures in the past two years.

Relatively few have left the search operation, however. And Google continues to hire engineers for it. "We're still investing a lot in search," says Google CEO Eric Schmidt in an interview. "We are first and foremost a search company." To explore Google's prospects for staying ahead, BusinessWeek recently got a close look at the highly secretive search quality group. Its mathematical formulas and methods are closely guarded to stymie competitors and prevent spammers from gaming the system. But Google provided candid insights into how it's trying to make search so good that people won't bother with extended detours to Twitter, Facebook, and beyond.

The group's home is in Building 43 in the center of the sprawling Googleplex in Mountain View. Deliberately casual, the first floor features such Googley touches as a refrigerator case with free Odwalla juices and huge panels of whiteboards scrawled with product ideas and jokes ("Why are they called apartments if they're stuck together?"). The company's trademark primary colors are splashed everywhere, down to the bulletin-board pushpins. A huge pile of Legos is scattered at the top of the stairs for engineers who need a break from coding software. In a hallway, someone snoozes in a MetroNaps EnergyPod, a semiprivate chair module for catnaps. Much of the second floor, which houses many members of the search quality team, resembles a well-used playroom for a large family of overgrown kids, which is more or less what it is.

The team is composed of several hundred engineers, many from outside the U.S. and some with 20 years of experience in search and information retrieval technologies that predate the Web. In Google's version of Silicon Valley egalitarianism, most people, including executives, share cramped glass offices. They're crowded with multiple large computer monitors, along with whiteboards and yellow sticky notes covered with formulas and notes on arcane aspects of search technology. The office of Matt Cutts, head of the anti-spam unit, and four others bears an inscription above the door: "What could possibly go wrong?"

It's a question Manber and his team contemplate every day. Google can claim a number of search breakthroughs in its 11 years, starting with the co-founders' breakthrough PageRank formula, which gives higher ratings to pages many other sites link to, because they're likely more useful than pages with few links. But it's clear these guys—and a few women—view doing search well as something as difficult as curing cancer. "There's just an endless supply of very difficult, challenging, but incredibly interesting problems to work on," says Scott Huffman, who runs the search evaluation unit and mobile search.

Lately Google has released a flurry of features that suggest it's paying close attention to the competition. One, called Search Options, opens up a pane on the left side of the Google page that allows searchers to narrow queries by such categories as videos, books, or time. In what may be a nod to Twitter, users can select the past 24 hours or even more recent results. Another feature, Google Squared, organizes information on topics, such as dog breeds, into a table with descriptions, photos, and more—echoing Wolfram Alpha.

Most of the search quality group's contributions are less visible because its work is focused mostly on the underlying algorithms, the mathematical formulas that determine which results appear in response to a particular query. Google conducts some 5,000 experiments annually on those formulas and makes up to 500 changes a year. Some are as tiny as improving the results returned on queries such as "GM" by inferring whether someone is looking for the car company or sites on genetically modified food. Others—such as Universal Search, which two years ago added images, videos, and maps to search results that once were all just Web pages—are radical changes. "The core relevance of the results underneath is still the most important thing," says search expert Danny Sullivan, editor of the Search Engine Land Web site.

If Google's corporate mission is to "organize the world's information and make it universally accessible and useful," Manber articulates a similarly expansive vision of search. He notes that while the 20th century was all about conquering nature, the 21st century will be about understanding people—not just what they say they're looking for, but inferring what they mean from the most minute behavioral clues. "Search is a big part of it, possibly leading the way," says Manber, an intense but soft-spoken man who plays 10 different musical instruments strictly for his own enjoyment. "Our job is to do rocket science that will be taken for granted."

The group's work often begins with a complaint. Google users can flag a bum result by clicking a link at the bottom of results. In some cases, data collected from a standard set of sample queries that Google's computers are constantly running may indicate people aren't clicking on some results as much or in the same way as they did historically. Most ideas, though, come from Google engineers with an inspiration. At the Tuesday morning meeting of search leaders, briefings from colleagues all around Google may spark a new idea to improve a set of searches. Not least, engineers just keep their eyes open. While he browses local art shows, Cutts scribbles obscure artists' Web sites in a small notebook so he can check later on how well they show up in a search—or don't.

Whatever the source, "broken" queries and other proposals for improvements land in the e-mail box of Amit Singhal. The genial, 41-year-old Google Fellow (the company title for its most accomplished engineers) heads the core ranking team. Its job is to provide the most relevant links to pages, videos, and other information for every query and adjust the algorithms when results don't seem to match the queries. Making improvements follows a regular regime. Engineers on the team can essentially try out proposed changes on one of Google's copies of the entire Web, stashed on the company's massive network of computers. Once they think they have a fix, they send the proposal to the evaluation unit, "eval" for short. Those changes can affect many other queries in unexpected ways, so it's up to eval to run more rigorous tests to make sure a change is positive overall and doesn't hurt other results too much.

About three years ago, for instance, engineers proposed a ranking update that would include more results related to synonyms of the words or phrases a searcher typed in—theoretically providing more relevant results in some cases. A broad sample of queries seemed to indicate people on average liked the results. But when eval statisticians dug into results in particular countries, they ran into a big problem. In Chinese, terms such as "big school" and "little school" were coming back as synonyms, producing results that were "really bad, like this would be hugely embarrassing if we launch this," recalls Huffman, head of the evaluation unit. So despite the positive results in other languages, engineers had to try again. Ultimately, they found a bug in the software code that they fixed.

FASTER THAN AN EYEBLINK

That experiment points up another surprising aspect of Google's testing. Google is famous for its algorithms and data-driven approach, but the company depends nearly as much on a global network of human evaluators, or "raters." These part-time contract workers are asked to provide opinions on proposed changes and whether results are more relevant, among other things. Often their opinions carry the day. A couple of years ago, an engineer proposed that Google extract addresses from pages and display them, and perhaps a related map, in appropriate situations, such as when someone does a search for "MOMA New York." The raters liked it, and after a live test on Google confirmed its popularity, the feature was rolled out. About 10 changes are approved to go live at each Thursday morning launch meeting.

To an outsider, many of the changes may look impossibly trivial. Several years ago, for instance, engineers noticed that while Google was returning useful pages when someone typed an acronym such as "CIA"—providing links to the government agency and to the Culinary Institute of America—people were taking a slightly longer time than expected to click on one of them. So on the results pages, Google began highlighting in bold the full names. Immediately, Google saw more clicks through to pages—and faster, too. How much faster? Perhaps 30 or 40 thousandths of a second, on average, Singhal says. That's one tenth the speed of an eyeblink. "This was a small idea," concedes Singhal. "But we have a real responsibility as a company to respect people's time."

More recently, Google has been focusing on providing results more relevant to specific regions around the world. Engineers realized that people in India searching for "bank" didn't much care about Bank of America (BAC), even if it was in the news. So Google has been tweaking algorithms to emphasize the searcher's apparent location. Now, a search on "bank" on the U.S. site, google.com, will bring up links to Bank of America and Wells Fargo (WFC), while the same search on google.co.in, Google's Indian site, will bring up homegrown HDFC Bank (HDB) and ICICI Bank (IBN). It's one small reason Google has even higher market share in many other countries than in the U.S.—such as 88% in India, according to comScore (SCOR).

Making all these search improvement efforts even more difficult is the need to cull Web spam. Marketers of male enhancement drugs or purveyors of damaging software, for instance, are continually trying to fool Google's algorithms into ranking their pages up high. Cutts and his team are constantly on the lookout for ways to recognize them and squelch their appearance in results. In other cases, Webmasters who oversee sites run afoul of such efforts by accident. So Cutts, a voluble 37-year-old who currently sports a shaved head—the result of a lost bet that his team couldn't finish a project on a tight deadline—also educates Webmasters through conference appearances and informational videos on YouTube.

Cutts' unusually public role was on display at the recent Search Engine Strategies conference in San Jose. In this crowd, Cutts is a rock star, his blog posts studied as carefully as Wall Street traders deconstruct comments by Federal Reserve Chairman Ben Bernanke. Cutts served on a panel called "Extreme Makeover: Live Site Clinic," where three dozen Webmasters rushed up to provide business cards. The first site was a sex toys emporium called mypleasure.com. An unruffled Cutts suggested changing its URLs, or Web addresses, to contain more product-related words. Examining a Midwestern department store site, Cutts chided, "Your URL structure is pretty much a search engine obstacle course," advising the Webmaster to excise question marks and other symbols.

The reason Google has become such a powerhouse isn't just because its search technology is exceptional, of course. It's because the company perfected a way to match advertisements to its search results. Hundreds of thousands of advertisers in its AdWords program bid in an online auction to buy "keywords," or terms likely to be in search queries, that they hope will trigger their ads to run on the right side or the top of the search results page. It's a good bet that people searching on, say, "sony cybershot" are in the market to buy a digital camera, and retailers that buy that term are more likely to attract clicks and sales. Spending on search advertising has soared in recent years because advertisers tend to get strong returns for their money.

How well are all the search improvements underlying those lucrative ads working? While no one knows for sure whose search results are best, a number of independent experts continue to give Google the nod over its rivals. And they think the current crop of would-be disrupters isn't going to beat Google at its own game anytime soon. "None of these are big challenges for Google," says search expert Sullivan. "I think Google is still better in quality."

For now, at least. But Google engineers know they need to think outside the search box to stay ahead. A recent contest called CSI, for "Crazy Search Ideas," asked engineers to submit improvements they thought would never be approved because they were weird or seemed too minor. Some 118 entries were culled to four, which are still being explored. Describing such left-field efforts after the recent Tuesday morning meeting, Manber mused about how easy it is to climb a hill and think you're on top of the world. "My worry is we could be stuck on top of a hill," he says, "and it's not the right hill."