Two first-half goals were enough to see off their second Premier League side this season after they ousted Bolton in the previous round.
United, who are currently a disappointing 14th in the Championship table, came close to an eighth minute lead when youngster Luton Shelton flighted in a testing cross and Jon Stead headed down but keeper Joe Hart dropped smartly to save.
Four minutes later, the home side were in front with a little assistance from Manchester City fans.
Lee Martin did well to drive in a dangerous cross that caused problems and Michael Ball appeared to be distracted as the ball twice hit pale blue balloons that had drifted on to the pitch from the 6,000-strong travelling fans.
But Shelton kept his eye firmly on the ball and cashed in by placing his shot out of the reach of Hart.
An angry Hart then spent several seconds stamping on the remaining balloons to prevent a repeat.
A minute later City should have been level when Ball's raking cross found Emile Mpenza, but he failed to hit the target and headed over.
As City turned up the pressure for a leveller, Elano found space on the edge of the area and squeezed a low shot through a crowd of players but it grazed the post before going out.
It was the home side who struck again with a second goal after 24 minutes.
Derek Geary crossed from the right and, as City's defence dithered and failed to clear, Stead kept cool and rammed in his low shot.
City boss Sven-Goran Eriksson brought on teenager Danny Sturridge at the start of the second half and it had immediate results.
Martin Petrov's corner had the home side in trouble and, after the ball deflected to Sturridge 12 yards out, he crashed in a left-foot shot for his first City goal..
City hurled everything at the Blades in the dying minutes but the Steel City side held out.
Hereford United's brave fightback proved too little too late as Cardiff City moved into the fifth round.
Star man for the Bluebirds was full-back Kevin McNaughton, who scored the first and earned a penalty for Steven Thompson to extend the lead before Theo Robinson gave Hereford hope.
Manchester United's dream of a second treble in their history remains alive, but a 3-1 victory flattered them against Tottenham at Old Trafford.