Everyday, I get up and look through the Forbes list of the richest people in America … If I’m not there, I go to work.

Click here now to more: Personal Finance Education

Mortgage rates scream buy, but who is listening?

An odd scene has been playing out lately in the offices of mortgage brokers and bankers around the country.

Mortgage rates have sunk to levels not seen in more than a half-century – a seductive 4.58 percent for an average 30-year fixed loan. Yet brokers and lenders report not a flood but a trickle of customers.

So what’s going on?

Call it a tale of the haves and have-nots.

The haves are those who stand to save money from refinancing and have the financial standing to do so. Since mortgage rates have been low for so long, most of them already have refinanced in the past 18 months. Doing so again wouldn’t be worth the cost for most.

The have-nots? Those are the millions of Americans pummeled by the housing collapse. They have little or no home equity or no money for down payments. Or they lack the credit or steady income to get or refinance a mortgage.

The result is that brokers like Ginny Ferguson are filling their days doing something other than handling a stampede of customers buying homes or refinancing.

Ferguson, CEO of Heritage Valley Mortgage in Pleasanton Calif., has managed to stay busy: She’s archiving files, reviewing marketing plans and calling previous clients and agents to try to drum up business.

“Am I sitting around playing Solitaire on my computer? No,” she says.

The 4.58 percent average for a 30-year fixed-rate loan last week was the lowest on records that mortgage company Freddie Mac has kept since 1971. The last time rates were lower was the 1950s, when most long-term home loans lasted just 20 or 25 years.

Under normal circumstances, 4.58 percent would be irresistible. A decade ago, if you’d told David Christensen, owner of Mountain Lake Mortgage in Lakeside, Mont., that rates would drop this low, he wouldn’t have believed you. And if rates did somehow fall this far, he never thought he would lack for customers, as he does now.

Yet both have come true.

Christensen argues that mortgage lending standards have tightened so much since the financial crisis that many people with decent but not-stellar credit can’t qualify. Lenders are demanding stronger credit scores and higher down payments or home equity.

“The pendulum has swung too far the other way,” Christensen said. “It needs to come back to the middle.”

Overall lending has ticked up in recent weeks, driven by borrowers looking to refinance. But it remains only about half the level of early 2009.

Stricter lending rules aren’t the only factors behind the restrained demand. A tax credit for home buyers that helped lift home sales expired April 30. The result is that fewer people are taking out loans to buy homes.

And some borrowers who do have good credit and solid jobs are still being rejected for refinanced loans. It’s because their homes are worth less than they owe on their mortgage. They’re “under water,” in real estate parlance. About a quarter of American households with a mortgage are in this predicament.

Blame the housing bust. It shrank home values and depleted home equity.

Most people in the lending industry acknowledge that lending standards were far too lax during the boom. Yet these days, some brokers recall the boom times with a tinge of nostalgia. Buyers and refinancers were everywhere. And yet rates were higher than they are now.

In the summer of 2005, lending activity was about 30 percent more than it is today. And homebuyers and refinancers had to pay about a full percentage point more for a mortgage than today’s 4.58 percent.

“If the money was as easy as it was three or four years ago, I’d be the richest guy in town,” says Joe Bell, a mortgage broker and real estate agent in St. Petersburg, Fla.

Now?

“The phone rings a lot, but a lot of people can’t qualify.”

Part of the problem is that people have been able to receive mortgage rates under 5 percent at several points over the past 15 months. For them, spending thousands on fees to take out a new loan wouldn’t make sense.

For many of the homeowners who refinanced over the past two years, rates would need to drop to around 4 percent for refinancing to be financially worthwhile, said Patrick Cunningham of Home Savings and Trust Mortgage in Fairfax, Va.

“We’re turning down a number of people for every one person that we can get through,” Cunningham says. “That part is frustrating for us, certainly. I would say it’s even more frustrating for the consumer.”

The drop in rates this spring and summer has been a surprise. Mortgage rates had been expected to rise after the Federal Reserve ended its program to lower rates by buying up mortgage-backed securities.

At the start of April, rates started to rise. Good economic news had caused long-term U.S. Treasury bonds, a safe haven during the recession, to lose some appeal. As demand for Treasurys fell, their yields rose. And so did mortgage rates, which track the yields on long-term Treasurys.

But then several European countries fell into crisis over their debt burdens. Investors rushed back into the safety of Treasury bonds. That drove down Treasury yields – and mortgage rates.

The costs of refinancing are generally considered worthwhile for homeowners who can shave at least three-quarters of a percentage point off their rate and plan to stay in their homes for several years.

For mortgage lenders and brokers, refinancing clients are generally people with excellent credit, stable jobs and plenty of equity in their homes.

People like Chris O’Donnell, 43, of Centreville, Va.

He and his wife are on track to close their refinanced loan this month. They are pulling money out to buy a new heating and air conditioning system for a home they bought last year.

But they’re able to do so only because they had put down 50 percent of the purchase price when they bought the home. Few can afford to do that.

O’Donnell is shaving his mortgage rate by about half a percentage point to just over 4.6 percent. He’ll save about $100 a month on payments. But he notes the main reason he can do that is the economy’s feeble state.

“It’s good for us,” he said. “But it scares the heck out of me for the economy.”

By ALAN ZIBEL and ALEX VEIGA, AP Real Estate Writers. Zibel reported from Washington, Veiga from Los Angeles.