Cosby settled a sexual-abuse lawsuit brought by a former Temple University employee for undisclosed terms in 2006.
"Cosby to black America is an icon, but once an icon figure betrays the trust of the community, we have to withdraw our support and condemn their actions," said Najee Ali, who was among those calling for the resignation. "If they don't remove that star, we can call it the walk of shame."
Los Angeles Urban Policy Roundtable president Earl Ofari Hutchinson echoed his sentiment.
"We were there; we remained silent; but no more," Hutchinson said.
The chamber said it had no plans to remove Cosby's star, and has never removed one before.
Bill Cosby Deposition Reveals He Gave Women Quaaludes
"Once a star has been added to the Walk, it is considered a part of the historic fabric of the Hollywood Walk of Fame. Because of this, we have never removed a star from the Walk," Chamber of Commerce president Leron Gubler said in a statement.
Cosby received his star in November 1977, and it's placed in a prominent spot near the busy intersection of Hollywood Boulevard and North Highland Avenue. In 2014, vandals wrote "rapist" three times on the star following allegations of sexual misconduct against the comedian.
Visitors at the iconic walk had mixed responses about Cosby's star.
"You'd have to remove almost everybody's if you were concerned about morals. This is about what they've done in Hollywood, what they've done to entertain people," said Jason Young, an accountant visiting from Maryland. "He is still a star whether you like how he was or not."
Tricia Savino, a tourist from Singapore, posed for a photo next to Cosby's star with both thumbs down.
Bill Cosby Deposition Changes Jill Scott's Mind: 'I Was Wrong'
She said the star should be removed because Cosby doesn't deserve it anymore.
"It's kinda like celebrating his legacy but his legacy now is something else altogether," she said.
Earlier this week, Disney World Resort near Orlando, Florida, removed Bill Cosby's statue from its theme park.
A sexual awareness group has petitioned the White House to revoke Cosby's Presidential Medal of Freedom awarded to him in 2002 by President George W. Bush.
More than two dozen women have accused Cosby of sexual misconduct in the past four decades, and many alleged that he drugged and raped them. The 77-year-old comedian, who has never been charged with a crime, denied some accusations while declining to comment or respond to the others.