NY media reported that 53-year-old Cali was shot multiple times outside his auto on Staten Island on Wednesday evening, local time, by an unidentified man who then fled the scene in a blue vehicle.
Shortly after, emergency responders arrived on the scene and Cali, who was born in New York City, was transported to Staten Island University North Hospital, where he was pronounced dead.
No arrests have been made.
Cali is the first reputed crime family boss to be shot since the execution of former Gambino head Paul Castellano in 1985.
The Sicily native, who is married to John Gambino's niece, served on the family's ruling panel for several years before being promoted to acting boss in 2015, replacing the previous leader Domenico Cefalu, also known as "Greaseball".
Infamous mob boss Francesco "Franky Boy" Cali has died.
Rudolph W. Giuliani, who at the time was the United States Attorney for the Southern District of NY, condemned the killings and vowed to find those responsible.
The Genovese, Gambino, Lucchese, Colombo and Bonanno mafia families are believed to have controlled organised crime in NY for decades. The house is listed under the name of Rosaria Inzerillo.
Francesco Cali, 53, was found with multiple gunshot wounds to his body outside his home on Staten Island, according to the New York Police Department.
According to local media, Cali ascended through the ranks, becoming a powerful capo of organised crime before turning 40 - less than 10 years after becoming an inducted member.
Prosecutors claimed that Cali was the Gambinos' ambassador to the Sicilian mobsters and the go-between for the two organisations.
The Gambino crime operation is one of the five historic Italian-American Mafia families in New York, Reuters has previously reported, and it makes money through violence and extortion.