FERGUSON, Missouri (CNN) -

The tide may be turning in Ferguson, Missouri.

For the first time since a white police officer fatally shot unarmed black teenager Michael Brown, the streets emptied early Thursday.

Protest crowds thinned from hundreds to dozens late Wednesday, then disappeared. Thunderstorms may have doused their numbers.

"I hope that people do calm down and just continue it to be peaceful. I hope the outsiders leave and let the real individuals who are really trying to make a difference remain so that it can stay peaceful, so some type of resolution and change can happen," said Charles Davis, owner of a local burger bar.

Late Wednesday, police made targeted arrests. Earlier in the day, they suspended an officer caught on video pointing a semiautomatic rifle at peaceful protesters and shouting profanities at them the night before.

But community leaders were grateful that tensions on the streets had eased.

In a move that reflected what appeared to be increasing calm, Gov. Jay Nixon ordered the Missouri National Guard to begin withdrawing from the city. Troops had been sent there Monday to protect the police command center, which officials said had come under threat of attack.

"We're going to have a systematic drawdown. We're working with the commanders to do that, but we're going to make sure we keep safety there," the governor told CNN's Don Lemon on Thursday.

"As we see folks getting calmer, fewer arrests, fewer problems here, that mission we're going to draw down off that. We don't need the same force of strength," he said.

But even then -- when calm returns to Ferguson -- work in the community is not done, Missouri State Highway Patrol Capt. Ron Johnson told residents. He said police need to be educated, not by professors, but by regular people about how best to build bridges.

"We need not sit there and make excuses; we just need to make it right," he said.

We stand with Ferguson, Holder says

U.S. Attorney General Eric Holder spoke Thursday about his visit to Ferguson to check in on the federal civil rights investigation into Brown's shooting.

"This attorney general and this Department of Justice stand with the people of Ferguson," he told reporters in Washington.

Holder has assigned scores of FBI agents and Justice Department investigators to look into Brown's August 9 death in the suburban St. Louis city.

Police have said Brown and the officer, Darren Wilson, struggled over the officer's gun; witnesses who have spoken publicly say the 18-year-old had his hands in the air when he was shot.

On Thursday, a source with detailed knowledge of the investigation told CNN that reports in the media that Wilson suffered a fractured eye socket from his reported scuffle with Brown are false.

The officer had a swollen face and went to the hospital, where he underwent medical tests and was treated for his injury, the source said.

A St. Louis County grand jury began hearing testimony in the case Wednesday, but it's not expected to make a decision on charges against Wilson before mid-October.

While Holder said agents have made "significant progress" in their investigation, he said it will still take time to complete and asked for the community's support.

Holder met with Brown's family not long after the teen's mother viewed her son's bullet-riddled body at a morgue for the first time. He heard from a woman who said her brother died in an encounter with Ferguson police in 2011. And he spoke to the community of a festering mutual mistrust between law enforcement and many communities in the United States.

Brown's mother, Lesley McSpadden, told CNN's Anderson Cooper in an interview Thursday that Holder's visit made a difference.

"Because, you can read a person," she said. "And when you're looking at them and they're looking at you in your eyes, it puts some trust back there," she said.

His trip was not universally well-received, however.

Activist Akbar Muhammed gave the attorney general low grades for the trip.