(CNN) -

Say "urban poor," and the image that most likely comes to mind is one of young black men caught up in a swirl of drugs and violence and irresponsible single women having babies. But this pervasive stereotype overlooks a surprising reality: Many whites live side by side African-Americans in some of the country's poorest urban neighborhoods.

Because white poverty is less expected, less recognized and less studied, we often exclude poor whites from our discussions. That masks a fundamental truth about economic inequality: Poverty is colorblind. But neither is it the same for everyone, as the white poor benefit from a lifetime of the hidden perks of white privilege.

As our nation continues down the road of economic recovery, this is a reality our local and national policymakers cannot afford to ignore as they seek to address employment and income inequality.

We traced the experience of nearly 800 children in Baltimore for more than 25 years, from the time they entered first grade in the fall of 1982 in 20 Baltimore public schools to well into their third decade. Half their families were low income, according to school records, and the typical low-income parent hadn't finished high school. What might be surprising is that of that half, 40% are white.

Looking at where these children started in life and where they ended up, the study results are troubling but clear: At 28, hardly any of the children from a disadvantaged background, black or white, had finished college.

But even without the benefit of a college degree, whites, and white men especially, had vastly better employment outcomes. At every age, the white men experienced shorter spells of unemployment, were more likely to be working full-time and earned more.

Baltimore, like so many other American cities, suffered immensely under the ravages associated with de-industrialization: the loss of industry, population and wealth. Under such circumstances, many of the city's disadvantaged youths stumbled along the way.

But the consequences have been especially dire for African-Americans. As young adults, African-American men had fared much worse than whites in the job market, even though they and their white counterparts had about the same levels of education and the whites reported higher rates of marijuana and heavy drug use and binge drinking.

Take, for example, the types of jobs the men in our study held. At 28, nearly half of the white men who had not attended college were employed in the industrial and construction trades, the highest-paying sector of blue-collar employment. By contrast, only 15% of African-American men worked in these sectors, and even within that small group, annual earnings were less than half that of whites -- $21,500 versus $43,000.

This disparity is no accident.

It fits a broader pattern evident as far back as high school: About one-fifth of white men who grew up in disadvantaged families had after-school and summer jobs in these industries -- important experience that can help secure a full-time job -- while not a single African-American person did.

Indeed, throughout the course of our study, it was clear that African- Americans face greater barriers to employment. Having an arrest record or failing to complete high school were less consequential for white men than for African-American men: 84% of whites without a high school degree were employed at 22; among African Americans, just 40% were.

Racial inequality also is embedded in hidden ways in other spheres of life, including discrimination in housing and banking practices that have kept white and black Baltimore substantially separate and cut off working class African-Americans from potentially valuable social contacts.

Why do differences in employment track so sharply with color lines?

The race-based privilege that benefits working-class whites over working-class African-Americans has its origins in the discriminatory practices that excluded African-Americans from the skilled trades during Baltimore's booming World War II and post-war industrial economy.

Although overt racial discrimination has lessened since then, the deep structural inequalities these barriers helped establish continue today through word-of-mouth hiring, employer attitudes that limit opportunities for African-Americans and segregated social networks.

The differences in how these young people found jobs illustrate the invisible ways race-based privilege is institutionalized in the job market.

When asked at age 22 how they found their current jobs, whites more often mentioned help from family and friends, while more African-Americans found jobs "on their own." The white job seekers in our study had family, friends and neighbors who could help them access good-quality, higher-paying jobs.

And what of those women having babies?

Most of the women of disadvantaged background, white and African-American, became mothers as teenagers, worked sporadically and when working, their employment was concentrated in the low-pay clerical and service sectors.

The difference, though, is that many more white women were married or in a stable co-habiting relationship. An additional earner in the household makes a vast difference in economic well-being, which means that white men's workplace advantages benefit white women as well.

As Americans, we like to think that we are all on a level playing field. Our society treasures rags-to-riches stories of individuals overcoming their humble origins to achieve the American Dream. But, the harsh reality we witnessed in Baltimore is that race and class place severe limitations on a child's ability to achieve that dream.

Too often, our policymakers focus on colorblind solutions, such as the Earned Income Tax Credit, to help the urban poor. Such programs only help those who already have jobs and fail to address chronic unemployment among African-Americans.

Amid the growing national conversation on economic inequality, now is the time for our leaders to recognize that race matters and develop creative programs, such as President Barack Obama's "My Brother's Keeper" initiative, to address the different challenges facing poor African-Americans.

Tracking the lives of Baltimore children for 25 years, we witnessed all too clearly how family conditions and poverty early in life cast a shadow that follows children into adulthood and how that shadow extends much further if you are African-American.