He was 79.
Rocco died on Saturday at his home in the Studio City neighbourhood of Los Angeles.
The cause was cancer, his manager, Susan Zachary, said, reported New York Times.
Rocco had fairly limited screen time in The Godfather (1972), but he emerged from that film with a collection of signature lines, including "You don't buy me out. I buy you out" and "Do you know who I am?" (both spoken to the Godfather-in-waiting, played by Al Pacino), and a Hollywood reputation for stealing scenes with little more than a Boston attitude and his eyebrows.
In 1990 he won an Emmy Award for his role as a larger-than-life old-school talent agent in the well-reviewed but short-lived Jon Cryer sitcom The Famous Teddy Z.
Rocco's other noteworthy films included The Friends of Eddie Coyle (1973), with Robert Mitchum; Freebie and the Bean (1974), one of several projects he did with Alan Arkin; Tom Hanks's That Thing You Do!" (1996), as a fast-talking music executive; The Wedding Planner (2001), as Jennifer Lopez's old-fashioned father; and A Bug's Life (1998), as the voice of the grumpy grain-counting ant Thorny.
(He once said of his voice work, which also included the role of a cynical cartoon producer on The Simpsons, "It's like stealing money.")
Rocco, whose real name was Alexander Federico Petricone Jr, was a Leap Year baby, born in Cambridge, Massachusetts, on February 29, 1936, to Alexander Sr and the former Mary Di Biase.
Rocco moved to Southern California in the early 1960s and worked as a bartender while studying acting with Leonard Nimoy.
His first film role was in Motorpsycho! (1965), a Russ Meyer special in which he played a biker-gang
PIX: Arjun Kapoor, Kangana, Ranveer party with Priyanka Chopra
Kareena Kapoor's Biggest Blockbusters
8 life lessons Game of Thrones teaches you
Pune and its ghosts
Army General warns of new tragedy in Kashmir