sharp struck in the 41st minute, collecting a free-kick from Jim Goodwin on the edge of the box and turning to find the bottom far corner of the net with a well-struck shot.
That was the moment that settled a tough top of the table battle on a difficult, bumpy pitch.
City had the edge in the early stages, forcing a number of corners and center-back Louis Carey went close with a header that flew just inches wide.
Winger Scott Murray might also have done better with a fierce left-foot shot that flashed wide of the far post.
Scunthorpe fought their way into the game with some good work by winger Kevan Hurst, making his debut on loan from Sheffield United.
But Sharp was always the dangerman to the visitors with his movement in and around the box.
At the start of the second half striker Jermaine Beckford had a chance to stretch Scunthorpe's lead, but his touch let him down at the far post with only goalkeeper Adriano Basso to beat.
Midfield man Lee Johnson kept driving City forward but they struggled to break down a solid Scunthorpe defence in which centre-backs Andy Crosby and Steve Foster were commanding figures.
Bristol City created just one good chance to equalise in the 75th minute when striker Kevin Betsy steered a close-range shot just wide of the near post with the goal seemingly at his mercy.
That was the last opening they created and in the closing stages it was Sharp who almost made it 2-0 when he volleyed just wide of the far post from an excellent cross by substitute Ian Morris.
Scunthorpe also went close again when a cross-cum-shot from Cleveland Taylor which was scrambled off the line by centre-back Jamie McCombe.
It was a huge win for Scunthorpe who are dreaming of Championship level football for the first time in 43 years.