Aaron Kirman‘s telling Josh Altman to slow his roll on the number of people leaving the Pacific Palisades … claiming his clients all want to rebuild — so, don’t expect the mass exodus others have talked about.

We caught up with the “Impossible Listing” star Friday … and, he says he saw the video of Josh telling us more than half the people in the Palisades are likely to abandon the neighborhood.

AK says he couldn’t disagree more … admitting a bunch of clients have reached out to him in the aftermath of the devastating Los Angeles wildfires — but, if he had to put a number on it, around 80% of those in the Palisades plan to rebuild.

Like a host of other real estate insiders we’ve talked to, Kirman isn’t sugarcoating the rebuilding process … saying it’ll take upwards of 10 years before the area is back to where it was.

For those people who don’t want to wait five to ten years to live in a new home, Kirman says the ones who can afford to buy are looking to buy in Beverly Hills, Santa Monica and other affluent regions of L.A.

However, Aaron says that’s all temporary … ’cause all his clients say they eventually want to go back to PP.

Kirman says a number of his clients are also P.O.’d at local L.A. officials … blaming their lack of preparedness for the tragedy. Like we’ve told you … Heidi Montag and Spencer Pratt are suing the city after losing their home in the Palisades.