Sherman joined the Red Hot Chili Peppers in 1983, replacing founding member Hillel Slovak, and played on the band's self-titled debut album. Sherman toured with the group in 1984 and co-wrote songs on its second album, Freaky Styley. He parted ways with the Chili Peppers in 1985 amid rising tension with frontman Anthony Kiedis and bassist Flea, and was replaced by Slovak.
Sherman's vocals later appeared tracks from the Red Hot Chili Peppers' 1989 album, Mother’s Milk, including a cover of the Stevie Wonder classic "Higher Ground."
Following his exit from the Red Hot Chili Peppers, Sherman played on albums by artists including Bob Dylan and George Clinton.
Despite his early contributions to the Chili Peppers, Sherman was not included when the band was inducted into the Rock and Roll Hall of Fame in 2012.
"It's really painful to see all this celebrating going on and be excluded," Sherman told Billboard in an exclusive interview at the time. "I'm not claiming that I've brought anything other to the band... but to have soldiered on under arduous conditions to try to make the thing work, and I think that's what you do in a job, looking back. And that's been dishonored. I'm being dishonored, and it sucks."
Gang of Four's Andy Gill, who produced the Chili Peppers' self-titled debut, told Billboard in 2012 that Sherman's role in the band should not be overlooked.
"I do find him to be significant to the band's history, very much part of getting the funk guitar in there. They just really rubbed each other up the wrong way," Gill said.
In his 2004 autobiography, Scar Tissue, Kiedis admitted that Sherman played a crucial role in building the band's successful career.
"God bless Jack, he did keep the band afloat for a year, and if he hadn't, the years to follow probably wouldn't have," Kiedis wrote.