That's where, on Wednesday night, police and the FBI arrested Irving as he, his wife and their grandchild were watching television.
"He requested to know how we found him," Maine State Police Sgt. Robert Burke told reporters.
Burke would not divulge what led authorities to Irving. On the front of the house on South Street, a sign reads "Irving." Police said Gary Irving had been living as Gregg Irving, the name of his younger brother. Records show he had lived there since 2002 and perhaps since the mid-1980s, police said.
Confirmation of his identity was made via a scar on his chest from heart surgery he underwent as a child and a fingerprint match, police said.
Investigators found guns in the house, police said. As a felon, Irving was not allowed to possess the guns, and he will be charged by federal authorities on firearms offenses, Massachusetts State Police spokesman Dave Procopio said.
When he showed up in court on Friday, dressed in a yellow jumpsuit with his wrists cuffed in front of him, Gary Irving looked little like his Most Wanted poster, which showed 1978 a picture of him clean-shaven.
In court, his hair was tied into a ponytail, his beard was bushy, and both were flecked with gray.
But one thing had not changed. "I'd know those eyes anywhere," Dickson said after looking at television accounts of her former friend. "Those are the eyes."
He was taken to Cumberland County Jail in Portland as a fugitive from justice, this time without bail.
"I can tell you that his wife appeared like she was in a state of shock," Burke said.
Neighbors in Gorham, a town of 16,000 residents, also expressed shock that the house with the swing set outside had been home for years to one of Massachusetts' most-wanted fugitives.
"Everybody's quite amazed at what took place here," said Keith Meggison, who has lived for 28 years on South Street, four houses down and across the street from the Irvings. "It was kinda strange that nobody picked up on this."
The Irvings apparently did take pains to keep out of the public eye. "Their windows were always closed; they always had sheets over their windows," said neighbor Alyssa Lurvey. "They kept to themselves so much that we never saw them, we never knew who they were."
"He was always pleasant and nice," neighbor Patricia Dixon said. "You never know who's living next door, I guess."
"We really are just starting to understand and know who this guy is," Maine State Police Lt. Walter Grzyb said. "They're going to want to know everything about him that has happened since 1978."
Police said they are planning to see whether they can link Irving to any unsolved rapes in the area. His house, on a main street into town, is about a quarter-mile away from three public schools and less than a mile from the University of Southern Maine. Some 60% of the 1,200 students who live on campus are women, a school spokesman said.
Dickson said she plans to visit her former friend in prison to ask him about his transformation 34 years ago: "What happened to you? What were you thinking? How did you get so angry?" she said she will ask.