n a warm spring day in April 1983, I stood across from what had been the United States Embassy in Beirut and watched as rescuers picked through tons of mangled steel and concrete littered with glass shards. Tenderly, emergency crews put salvaged bits of bodies in small blue plastic bags. Forensic experts later matched up the pieces so they could be buried together. More than sixty Americans were killed in that lunchtime bombing.

Some of the dead had been my friends.

The attack was the first by a Muslim suicide bomber against an American target—anywhere in the world. Over the next eighteen months, Islamic extremists blew up a second American embassy and the U.S. Marine peacekeeping compound in Lebanon. The Marine bombing is still the largest loss of American military life in a single incident since World War II, larger than any attack in Vietnam or Iraq. I heard those bombs thunder through Beirut too and again watched weeks of rescue efforts. The three terrorism spectaculars marked a turning point for the Middle East.

Since then, Islamic extremism has progressively grown into the most energetic force in the Middle East—and the gravest threat to Western interests. Out of my own sense of anguish, I tracked the trend as it unfolded in country after country and wrote one of the first books, in 1985, about this new form of sacred rage that has since redefined the world’s political divide. Fear of its explosive potential will define American foreign policy for years, potentially decades, to come. Al Qaeda and a growing array of offshoots continue to push their tentacles deeper throughout the region—and beyond.

Yet a generation later, Islamic extremism is no longer the most important, interesting, or dynamic force in the Middle East. The hard-core terrorists in al Qaeda or Islamic Jihad have repeatedly proven that they can destroy. But they have yet to provide tangible solutions or viable new models for problems plaguing the region.

In the early twenty-first century, a budding culture of change is instead imaginatively challenging the status quo—and even the extremists. New public voices, daring publications, and increasingly noisy protests across two dozen countries are giving shape to a vigorous, if disjointed, trend. It includes defiant judges in Cairo, rebel clerics in Tehran, satellite television station owners in Dubai, imaginative feminists in Rabat and the first female candidates in Kuwait, young techies in Jeddah, daring journalists in Beirut and Casablanca, and brave writers and businessmen in Damascus.

For all, peaceful empowerment has become the preferred means of making political decisions and producing change.

The tiny minority willing to go out and kill has had such impact in part because there have been so few other political ideas and activists in the region offering alternatives. Now, increasingly, there are. A trend struggling for decades to take root has finally begun—and, I stress, only begun—to have impact.

Impatience and frustration fueled by education, technology and the miracles of instant media, demographics, globalization, and change elsewhere in the world have altered the equation.

The mujahedeen, or holy warriors, have long dominated the headlines. Today, however, the so-called pyjamahedeen, or pajama warriors, are increasingly capturing the public imagination too. They campaign for change not with bombs on battlefields, but from laptops at home. Web sites and blogs have become the twenty-first-century chroniclers of police crackdowns, human-rights abuses, and election irregularities. In countries where I once sought out clandestine cells, I now also look for computer nerds. So, tellingly, are government security forces. Some of the pyjamahedeen have already gone to prison.

“Governments have a new kind of opponent,” said Egyptian blogger Wael Abbas. His blog has posted cell-phone videos of police brutality, including one of a detainee writhing in pain as he was sodomized by police with a broomstick. Started in 2004, Abbas’ blog was getting up to 30,000 hits a day—and up to 45,000 during a crisis—by 2007.

“We are not bound by government rules, like the political parties.We can use the language of freedom,” he told me. “We offer an alternative voice, especially for the young.”

The issue in the Middle East is no longer whether to engage in political transformation. The issue today is how to get there.

“In the Arab world, the status quo is not sustainable,” reflected Marwan Muasher, a former Jordanian foreign minister who became a World Bank vice president. “What worked forty years ago—when the state could decide things and expect people to follow—does not work now. Unless the state is responsive and aware, it is in for major trouble.”

Pressure is now mounting on virtually every Middle East regime. Even the conservative Gulf sheikhdoms have had to respond, albeit in the smallest feasible steps. Saudi Arabia held its first (male only) elections for local councils in 2005, while Kuwait’s parliament ended a long boycott and finally granted women the rights to vote and run for office in 2006.

“Regimes are increasingly unable to deliver what they promised or protect their people, and they certainly provide no direction for the future,” explained Nader Said, a Palestinian pollster and political sociologist. “Mix that with a constituency that is more demanding, and more aware, and more in search of rights.

“Most in the Arab world now think they deserve better.”

Regimes have been forced to adopt the language of democracy, whatever their real intentions or conniving to prevent it. The definition publicly embraced in the Middle East is the same as it is everywhere else, reflected in the Alexandria Statement produced at a meeting of 165 civil society leaders and government officials from eighteen Muslim countries at Egypt’s rebuilt Alexandria Library in 2004.

When we talk of democratic systems, we mean, without ambiguity, genune democracy. This may differ in form and shape from one country to another due to cultural or historical variations; but the essence of democracy remains the same. Democracy refers to a system where freedom is the paramount value that ensures actual sovereignty of the people, and government by the people through political pluralism, leading to transfer of power. Democracy is based in respect of . . . freedom of thought and expression and the right to organize under the umbrella of effective political institutions, with an elected legislature, an independent judiciary, a government that is subject to both constitutional and public accountability, and political parties of different intellectual and ideological orientations.

Few governments have begun yet to honor those words. What makes this era different are the activists now trying to hold them to account. They are no longer limited to the intellectual elite. And the numbers engaged—especially compared to the small cells of suicide bombers—are striking. Roughly one quarter of Lebanon’s entire population took to the streets, peacefully, in 2005 to demand the government’s resignation and an end to Syria’s twenty-nine-year military occupation. Despite dangers from an escalating insurgency, Iraqis poured out in three elections that made the purple ink-stained finger famous. Participation increased with each vote.

More than 100,000 people turned out in the Jordanian capital to protest after suicide bombers simultaneously struck three Amman hotels in 2005. “Burn in hell,” they shouted, in rhythmic unison, after the al Qaeda leader in Iraq claimed credit.

And none of these were rent-a-crowds, the usual means of producing mass turnouts in the region.

“These initial signs were intoxicating. They produced wonderment,” mused Ghassan Salameh, Lebanon’s former minister of culture. “A region long dead politically suddenly had a pulse.”

Violence is increasingly unacceptable to the majority, according to public opinion polls and petitions. In 2004, more than 2,000 Muslim intellectuals signed a petition calling on the United Nations to sponsor a new international treaty that would outlaw the use of religion to incite violence. It proposed that the Security Council create a new international tribunal to try “the theologians of terror” and the “sheikhs of death” who provide religious cover for terrorism. The petition also urged members of the world body to prevent broadcasts of “the mad musings of the theologians of terror.”

Even as people turn to political Islam, they are turning against Islamic extremism. Clerics and theologians have begun to challenge bin Laden and al Qaeda with their own fatwas in what has been dubbed the “counter-jihad.” Even in Iraq, some Sunni tribal leaders turned on al Qaeda cells after they went too far in kidnapping and killing the local populations.

Militant movements are under pressure too. A few of the groups that began as secretive cells have also begun to emerge from the underground to run for office. Motives are often suspect, but it is also striking that they are appealing to voters on platforms that deal with everyday issues such as better garbage collection, improved health care, and less corruption.

The new momentum has spurred talk of a nahda, Arabic for “awakening” or “renaissance.”

“Is it something real? Is this finally an Arab spring?” asked my old friend Saad Eddin Ibrahim, the leading human rights activist in Cairo, who was jailed three times and left crippled by prison abuse. “Our desert region is famous for its mirages. But these are real visions of change.

“The despots in the Arab world are on their last gasp,” he reflected. “Just like any last-ditch battles, they will do a lot of stupid things and leave a lot of destruction. But these will be the last battles. People have already broken the fear barrier. They are as ready for change and democracy as Eastern Europe was in the 1980s and as Latin America was in the 1970s. History is moving. The moment is ours.”

That’s the good news.

The so-called “Arab spring” of 2005, which offered greater promise than at any time since most countries gained independence, did not endure. It did not set off the toppling dominoes of regional change, as the fall of the Berlin Wall did in Eastern Europe. For much of the Middle East, the challenge of change is today tougher than anywhere else in the world.

The Middle East, concluded a United Nations survey in 2005, faces an “acute deficit of freedom and good governance.” Most Arabs live in a “black hole . . . in which nothing moves and from which nothing escapes.” The region has the largest proportion of ruling monarchies (eight) and family political dynasties (four) in the world.

The dangers of change are also already visible.

Democracy is about differences—and they are bound to flourish once disparate sides of society are really free for the first time to speak and make their own specific demands. Unity in opposition to tyranny almost never translates into unity once in power. In a region rife with vulnerable minorities and shifting demographics, opening up politics endangers deepening the problems it is meant to solve.

As Iraq has illustrated too vividly, democracy unleashes existential dilemmas.

Opening new space also does not guarantee what or who will fill it. Because the political debate in the Middle East is grounded in its own experience, the face of change will be too. More often than not, Islam will be the dominant idiom of opposition and change. Indeed, the region may not be transformed without tapping into its religious traditions—because of their appeal and legitimacy, but also by default.

Shortly before he was killed in 2005 by a bomb placed underneath his car in Beirut, Lebanese historian Samir Kassir opined in a little book entitled Being Arab,

Not only can the current regimes not give, or restore, to their states the ability to take the initiative in international affairs, they also forbid their citizens any license—­if not to change these regimes, then at least to breathe new life into them through popular participation. . . . The crisis of faith in the political process then runs its course, until there is nothing left but religion to channel people’s frustrations and express their demands for change.

Although today militant Islam appears primarily to target the West, it was initially a product of the impasse in Arab states. . . . The rise of political Islam took the form of a re-­Islamization of society in response to what were considered to be inefficient, iniquitous or impious governments, rather than a reaction to the culture of modernism.

The period of change will often witness an uneven contest pitting inexperienced democratic activists with limited resources against both well-heeled autocrats who have no intention of ceding control and Islamists who believe they have a mission from God and a flock of the faithful to tap into. It will be an unfair battle from the start.

Nothing will happen quickly, either. Even regimes that acknowledge the need to open up politically talk about gradual steps, in phases, over years or decades or generations.

“Change is a future notion,” reflected Marwan Muasher, the former Jordanian foreign minister. “The trick is putting it in the present.”

In 1995, Sheikh Hamad bin Khalifa al Thani of Qatar—the little emirate jutting off Saudi Arabia into the Persian Gulf—became the first of a younger generation of Gulf princes to assume power. He did it by overthrowing his father, the region’s most autocratic leader in what was then the Middle East’s most closed society. To the consternation of neighboring sheikhdoms, Sheikh Hamad then invited Israel to open a commercial office in Doha, the United States to headquarter Central Command in Qatar, and American and European universities, including Cornell and Georgetown, to open up Qatari branches. He also launched al Jazeera, the first all-news Arab satellite channel. All were bold, controversial moves. Yet Qatar’s emir is still pacing the spread of political participation.

“We have no intention of waving some magic wand and changing our entire culture and society overnight,” Sheikh Hamad told me.

“To hurry change would only invite the social instability we seek to avoid, so we have chosen a middle course for change. Compared to radical changes in other nations since the end of the Cold War, our changes might appear small, but they are well-planned.

We must be careful to change at a pace that meets the needs and desires of our people, as well as our traditional culture steeped in thousands of years of Arab and Islamic history.”

Qatar has held three municipal elections since 1999, conducted a referendum for the first constitution in 2003, and scheduled the first elections for a new national legislature in 2007. Women are able to vote and run for office.

Yet Qatar’s new legislature will have thirty elected seats with fifteen more appointed by the emir—still giving the palace effective veto power. The emir’s rule is still absolute.

“The Arab world is developing. Nothing can stop what is happening. But for now, the trend is toward participatory despotism,” said Harvard-educated Paul Salem, head of the Beirut branch of the Carnegie Endowment for International Peace and son of a former Lebanese foreign minister.

“Most of the Arab states are making that shift, and it’s important. That’s how it started in the West: The king allowed a parliament and eventually parliament got stronger. After that, the dynamic will be slow liberalization within authoritarian systems that accept and integrate, to varying degrees, the principle of participation in parliament, in local government, in open discussion of the budget, and with emerging press freedoms, nongovernment organizations, more political parties, and more human rights groups.

“It will happen,” Salem said, “but with much grinding of the wheels and sluggishness and tension.”

And tokenism. Elections engineered to produce token participation are preempting real democracy in several states.

In 2006, the seven sheikhdoms in the United Arab Emirates—the most opulent and liberal society on the Arabian Peninsula, complete with bikini beaches and restaurants serving alcohol and pork—held their first election. More than 400 candidates ran, including sixty-five women. The process, however, was a political charade. Only 6,595 people, all handpicked by the government out of 300,000 citizens over the age of eighteen, were allowed to vote. The stakes—twenty seats on a new Federal National Council—were also a token. Another twenty seats were appointed by the ruling tribal royal families. And the new council had no powers. It was only an advisory body that could be ignored. The first elected body was not a credible political entity in the twenty-first century, even as a transition device.

Several Middle East regimes, particularly the eight oil-rich members of OPEC, can literally afford tokenism. The wealth of so-called “rentier” states—which survive off the “rent” of foreign payments from petroleum products—can limit both public leverage and foreign pressure.

Petroleum and democracy do not mix well. “Tyranny has a full tank,” said Marina Ottaway of the Carnegie Endowment for International Peace. And the push for change began just as oil prices soared to unprecedented highs.

Petrodollar wealth can also breed political apathy in traditional societies.

In little Qatar, which has the world’s second-largest gas reserves, the emir worried that people with one of the world’s highest per capita incomes would not be interested in politics. Qatar’s stock market is instead the national obsession.

“To get people to accept a constitution or the idea of voting took us time. Some people didn’t see it as important,” Sheikh Hamad told me in Doha, the construction-crazed capital, in 2006. “Our society is tribal. It means that tribes select their eldest as their leader, and they just listen to his order and his wisdom. People are still comfortable with that system. They don’t see any reason to work in politics.”

The election turnout for the 2003 municipal election was 32 percent. In 2007, it was 51 percent. Only a minority of citizens, however, was registered to vote for either election.

As change does play out, the emerging governments will also be weaker and far from full democracies for a long, long time to come. New leaders will remain vulnerable.

“America is different from everywhere else. It was full-born in a constitution. The rest of the world had to work toward a constitution,” Paul Salem said. “Remember that when you make calculations about the Middle East.”

The disastrous miscalculations in Iraq will further slow and complicate the process. The American intervention was, in the end, often counterproductive to the cause of democracy in the region. It disillusioned many about both the costs and benefits.

All the factors contributing to change will also make the region susceptible to greater turmoil and divisions during the transitions. The stimulants pose hazards.

Among them, a major engine of change is youth. The Middle East—­including North Africa, the Levant, and the Persian Gulf—has witnessed a sevenfold explosion in population over only three generations, from 60 million in the 1930s to 415 million by 2006.11 The majority today are young. In some countries, up to seventy percent are under age thirty. They are hungry for change. As they come of working age, economists contend that they offer the potential for the kind of economic growth witnessed with the Asian Tigers. Many of the young are willing to act politically, too. And numbers are on their side.

Yet as governments are increasingly unable to provide education, employment, and housing, today’s frustrated youth are also looking for fast answers. Regionwide, roughly one in three young people is unemployed. The highest youth unemployment rate in the world also provides an exploitable flashpoint to build sympathy or support for extremist movements.

“Tens of millions of educated but underemployed, unemployed, restive and frustrated young men and women have given unnatural birth to thousands of active terrorists and anarchists, targeting our own and foreign lands,” wrote my colleague and friend Rami Khouri, a columnist for Lebanon’s Daily Star and head of the American University of Beirut’s think tank.

The many arms of information technology and the media have also had a huge impact, again to good and bad effect. The Internet, even with government censorship or restrictions, provides access to the outside world as never before. In some countries, up to seventy percent of the young were using the Internet on a weekly basis in 2006, surveys show. The cell phone text message has become the medium to organize antigovernment rallies. And satellite television has revolutionized a region where all media were state-controlled not so long ago.

After decades with only one or two heavily censored television channels, the first Arab satellite news station debuted in 1996, providing access for the first time to news not controlled by local governments in a language most in the region could understand. By 2007, dozens of unregulated satellite stations beamed in via rooftop satellite dishes began making people far more aware—of what is really happening in their own countries, of change happening in other parts of the world, and of international standards for political life.

“Arab media is in the midst of a highly dynamic transition, fuelled by the emergence of low-cost and accessible satellite broadcast technology,” a report by the United States Institute of Peace concluded in 2005.

It is the satellite channels that show the greatest potential for ushering in political change in the region. . . . No matter how hard they try, regimes can no longer control the information environment. . . . The fact that Arab governments call in to television [talk] programs to defend their human rights record indicates some advancement.

But technology’s wizardry also allows Osama bin Laden and his affiliates to distribute worldwide their diatribes, threats, and beheadings on satellite television, while terrorist groups tap into the resources and riches of the Internet. Propaganda has a new tool with unprecedented reach.

For now, Middle East societies are caught in an awkward stage in transition. In 2003, the Lebanese Broadcasting Corporation launched Star Academy, a mixture of American Idol and Big Brother that brings together Arab youth from all over the region to compete for a singing contract. An affiliate station tracks the competitors all day, every day, as they sing, dance, play, interact, cook, and sleep. Conservative clerics quickly condemned the show. Saudi Arabia’s Grand Mufti Abdul Aziz al Sheikh, the kingdom’s highest religious authority, issued an edict calling the show an open invitation to sin. He warned Muslims to avoid it. Nevertheless, it became wildly popular.

Finally, greater exposure to ideas in the outside world has helped inform and change public attitudes in the Middle East. Some of the most popular books in the region are written by or about Western politicians, philosophers, and political analysts. The osmosis of globalization has spurred a rich new discourse on democracy and how to adapt it to their own cultures.

Yet exposure has also alienated those fearing foreign encroachment— again. Many of the most ardent Iranian revolutionaries were educated in the United States; one group was often referred to as the “Berkeley mafia” because they all attended the University of California. In the very worst case, Mohammed Atta, who masterminded the terrorist spectaculars against the World Trade Center and the Pentagon on September 11, 2001, spent almost a decade in Europe.

Change and its agents will produce some troubling consequences. Many in the region will question whether the short-term dangers are worth the eventual benefits—and the price paid along the way.

The Middle East is not really one place, so change will have many faces.

The region today is arguably more stereotyped than any other part of the world. But the peoples, histories, religions, political systems, and economies actually differ widely among countries, even within them. It is the world’s only bloc spread across two continents.

“Democracy,” said Paul Salem, “will be a country-by-country phenomenon.”

The Middle East includes the tribal societies of the Arabian Peninsula, from where Islam and the Arabs originated. It includes the cosmopolitan cities of new Beirut and old Damascus. It includes Palestinians who have lived more than a half century in squalid refugee camps as well as Gulf princes who own multiple palaces because of oil found under the desert sands. It includes the desert-dwelling Berbers and Bedouin nomads who roam with their camel-hair tents across the Sahara, the Sinai, and the vast expanses of Arabia. And it includes Kurds, who are the world’s largest minority without a state. Although they are not Arabs, they have significant numbers in Iraq, Iran, Syria, and Turkey.

Political starting points vary just as deeply. Saudi Arabia has strict Sunni religious rule, while Iran is a Shiite-dominated Islamic republic with a constitution that draws on European law. Syria and Libya are secular states based on socialist ideologies. Kuwait, Jordan, and Morocco are still ruled by traditional monarchies.

The range of freedoms is reflected in the region’s fashions: In Arabia, the national dress for men is the traditional loose-fitting white thobe, which looks like a shirt that extends to the ankles; for women in public, it is a shapeless black cloak with four layers of black veils. In contrast, Lebanese men can wear tight Speedo briefs on mixed-gender beaches, while many women favor whatever fashion is the sassiest, flashiest, or skimpiest.

Economically, the peoples of the Middle East also have vastly different resources as tools for a transition. The region includes the earth’s richest nations, like glitzy Qatar, the tiny thumb off Saudi Arabia’s eastern coast that sits atop the world’s largest field of natural gas and has a per-capita income of $38,000. On the other extreme is exotic but densely populated Yemen on Saudi Arabia’s southern border, where the per capita income is a mere $500.

The broader Middle East is not even a single geographic unit. The two dozen nations spread from northern Africa to western Asia. They stretch from Morocco on the Atlantic Ocean to Lebanon and Syria on the Mediterranean, from Egypt and Yemen on opposite sides of the Red Sea, to Iraq and Iran at the top of the Persian Gulf and Oman at its mouth. The region spans four time zones and 4,000 miles from east to west.

Languages differ too. Using French and English, I once translated between Moroccans and Saudis who could not understand each other’s Arabic. Most Iranians speak Farsi and haughtily note their Indo-European rather than Arab roots. Kurds have their own language too.

Although the region is associated with Islam, it is rich with religious minorities. One out of every ten Egyptians is a Coptic Christian. About fifteen percent of the Palestinians belong to disparate Christian faiths. The region is also home to Alawites, Armenian Orthodox and Catholics, Baha’i, Chaldeans, Druze, Greek Orthodox, Maronites, and many others.

I once visited a fire temple of the Zoroastrians, who worship light as the symbol of a good and omnipotent God. Iran is the world center for the faith founded six centuries before Christianity. As the symbol of light, fires at the altars of their temples have burned continuously for centuries. Zoroastrian ideas about the devil, hell, a future savior, the struggle between good and evil ending with a day of judgment, the resurrection of the dead, and an afterlife had great impact on all monotheistic faiths, and even Buddhism.

The largest Jewish population in the Middle East outside of Israel is also in Iran, which still has kosher butchers, Jewish schools, synagogues, and a first-rate hospital favored by many of the ayatollahs. In Tehran, I have attended a Hebrew class for children as well as a Catholic service at which wine was served—with government permission, in a country that otherwise outlaws alcohol—as part of communion. Iran’s parliament has five especially reserved (and proportionate) seats for Jews, Christians, and Zoroastrians.

Because of the region’s diversity, rivalries and backstabbing can become bitter and intense. Middle East nations can also be miserly with each other.

On a 1981 trip to Libya, I covered Yasser Arafat’s quest for financial aid. In a bizarre scene, Libyan leader Moammar Qaddafi escorted a fatigues-clad Arafat through the opening of a new People’s Store in Tripoli, showing off French fashions, German toys, Italian appliances, and Japanese electronics. At a rooftop ceremony afterwards, the two men prayed, shared dates and goat’s milk, then exchanged gifts. Arafat gave Qaddafi an exquisite antique camel saddle. Qaddafi presented the Palestinian leader, who was still in exile, with a set of Samsonite luggage.

“That’s all he’s likely to get from Qaddafi,” complained one of Arafat’s aides as we watched from the sidelines. “Qaddafi has promised us millions but never delivered a single cent.”

More than two decades later, several Arab governments that had pledged billions to help rebuild Iraq had failed to pay up several years after making their commitments—­ even though Iraq’s instability was affecting them all.

Opening up political systems may spark further friction, at least in the short term, as the balance of power shifts both within countries and between them. People long excluded will now want their say too. Indeed, whatever the rhetoric, the greatest tension in the region may not be between Arabs and Israelis. Sunnis, who long monopolized power, are particularly apprehensive about the growing leverage of Shiites, Islam’s so-called second sect. In an interview in 2004, Jordan’s King Abdullah warned me about the danger of an emerging “Shiite arc,” or crescent, stretching from Iran through Iraq into Syria and Lebanon—the kind of domino theory that once scared the West about communism. The phrase set off a firestorm; it is still used to describe the region’s growing sectarian split.

The Middle East has already gone through enormous change.

In the twentieth century, three pivotal events redefined the region. The collapse of the Ottoman Empire, which dominated the Middle East for five centuries, redrew the map and gave birth to modern states after World War I. The creation of Israel in 1948 changed the region’s political dynamics and spawned the world’s longest conflict. And the 1979 Iranian revolution introduced Islam as an alternative political idiom. All three had spillover worldwide.

By coincidence, I first landed in the Middle East on October 6, 1973, arriving in Beirut during the chaotic outbreak of the fourth modern Middle East war. The Arabs had just launched a surprise attack on Israel. “Egyptian troops have crossed the Suez Canal,” an American tourist leaned over and whispered to me.

Oil was then only $3.12 per barrel—yes, barrel, not gallon—and the sheikhdoms of the Arabian Peninsula were considered poor developing countries. In Saudi Arabia, schools for girls had only been around for nine years and a single-channel television service for seven; both had been introduced over serious objections by conservative clergy. The strict Saudi version of Islam did not tolerate the human image in art or literature, much less on the small screen.

Iran was then one of two pillars of United States policy in the region. Some 40,000 Americans—military trainers and government advisers, businessmen, Peace Corps volunteers and tourists—passed through each year. It was spy heaven for the Central Intelligence Agency, which trained and worked closely with its Iranian counterpart. In Tehran in 1973, I stayed at the high-rise Hilton, which had just hosted a pageant of exotic and scantily clad beauties competing for the Miss Iran title.

Lebanon was the region’s playground, a cosmopolitan tourist haven of Mediterranean beaches and scenic ski slopes with decadent nighttime pleasures, casinos, and nightclubs. And Washington still had diplomatic relations with Baghdad, which was ruled by the Baath Party of an up-and-coming politician named Saddam Hussein.

During more than thirty years of living in and traveling to the region, I’ve witnessed extraordinary transformations of all kinds. I’ve covered Middle East wars as well as the first phase of peace between Egypt and Israel. In the 1970s, I covered the Shah of Iran as well as the revolution that ousted him. I covered Yasser Arafat as the world’s most notorious terrorist in the 1980s, as a signatory to a Palestinian-Israeli peace accord on the White House south lawn in the 1990s, and as the obstacle to a final peace in the 2000s. I covered Saddam Hussein’s war with Iran in the 1980s, and with Kuwait in the 1990s, and then flew into Baghdad weeks after he was ousted in 2003. Along the way, I’ve talked to thousands of people—on the streets and in palaces, in the bunkers of war zones and the tranquility of religious sites, on university campuses as well as in banks, shops, and factories, in newspaper offices and Internet cafes, in shantytown slums and luxurious villas, in the corridors of power and the back-alley hideouts of extremist cells.

For this book, I wandered across the region again to write about the crises of change that have begun to redefine the Middle East in the twenty-first century. There are many fine books about the Arab-Israeli conflict, Islamic extremism, the Iraq war, and Iran’s turmoil. I deliberately set out to look through a different prism. This is a book about disparate experiments with empowerment in the world’s most troubled region. My goal was to probe deep inside societies of the Middle East for the emerging ideas and players that are changing the political environment in ways that will unfold for decades to come.

The trauma of the 9/11 attacks and the Iraq war—and other turmoil that almost certainly lies ahead—makes understanding the Middle East even more important. The central flaw behind the American intervention in Iraq was ignorance of the country and its people. Policy was ill informed. The U.S. decision to go to war was often based on wishful thinking and armchair projections at great distance.

“The United States didn’t have relations with Iraq for well over a decade, so there were very few U.S. officials who had any direct experience of the country,” reflected Charles Duelfer, who served for seven years as a UN weapons inspector in Iraq and returned after the war for the United States to write the final report on Saddam’s weapons of mass destruction.

“The analysts who were making judgments, making assessments, didn’t have a tactile feel for Iraq,” he said. “Many of them had never even met a real Iraqi. Their reality came from a computer screen. Some policy makers made decisions not knowing—or caring—how catastrophically ignorant they were about Iraq. They had no concept of the ground truth of Iraqi politics and society. They had only a cartoonist’s understanding of Saddam. They did not know what made Iraq Iraq.”

The United States and its allies cannot afford to make that kind of mistake again. The Iraq war has carried costs—in lives, resources, and both human and political relationships—that are still incalculable.

This book reflects the voices in the region, not the pundits from afar. It covers the full spectrum of politics, not just the new democrats or heroes of change. I tracked down the heads of militant Palestinian and Lebanese movements as well as the inspiring activists of Egypt and Morocco and the reformers risking their lives in Syria and Iran. Some countries, like Saudi Arabia, I left out because the system has prevailed and the voices of change are not yet noisy enough. I also did not include Israel because it teems with political diversity and open debate. I deliberately selected a variety of countries experiencing the dynamic tension of change and used each to reflect a different slice of it.

Not all of the new actors will succeed. Some may only nudge the door ajar for those who follow. And others will be totally unacceptable to the West, creating tensions and diversions for both sides along the path of change.

I have seen up close how volatile the period ahead will be. When I started out on this latest journey, the region was full of dreams. As I finished it, serious shadows loomed in many places. I chose the title of this book from a lament by Mustafa Kemal Ataturk, the Islamic world’s great twentieth-century modernizer, who molded today’s Turkey from the ruins of the old Ottoman Empire. “Neither sentiment nor illusion must influence our policy. Away with dreams and shadows!” he said.“They have cost us dear in the past.”

The Middle East will, unquestionably, continue to spur anxiety, threaten security, and drain resources. Yet even as the war in Iraq steadily escalated and the United States was increasingly discredited on democracy promotion, the majority of the people in the Middle East still wanted the kind of political change that has swept the rest of the world over the past quarter century.

What is most inspiring is not the dreams the outside world has for the people of the Middle East. It is instead the aspirations and goals they have genuinely set for themselves.