Calls for protests came after Israeli police fired upon Palestinian worshippers inside the Al-Aqsa compound after afternoon Friday prayers, sparking mass panic and hysteria inside the holy site.
The Israeli military did not comment directly on the deaths but said about 7,000 Palestinian "rioters" threw rocks and rolled burning tyres at soldiers, and at the fence itself, at several locations along the border.
The Gaza health ministry said 154 Palestinians have been killed since the protests began March 30.
There was no immediate word of any violence in the mosque, whose older male worshippers said they had been allowed to exit after being searched.
An Israeli police spokesman said firecrackers and stones were lobbed at police "for an unknown reason".
The unrest coincides with the first anniversary of the al-Aqsa protests in July 2017 when tens of thousands of Palestinians prayed outside the compound for almost two weeks, protesting against the new metal detectors installed by Israeli authorities.
Israeli forces detained five Palestinian youths and are forcefully removing the Al-Aqsa security guards from the compound, preventing the guards from remaining on the premises to secure the area.
Israel and Hamas have fought three wars since 2008.
The Gaza Ministry of Health said that Ghazi Muhammad Abu Mustafa, 43, died of wounds he sustained with a live bullet in the head east of Khan Yunis city in the southern part of the Gaza Strip.
Israel says its lethal tactics are needed to prevent armed infiltrations and accuses Gaza's Islamist Hamas rulers of encouraging the disturbances to distract from their governance problems under an Israeli-Egyptian blockade.
While many foreign powers have censured Israel's handling of Gaza, the United States has echoed its blaming of Hamas.
Hamas has pledged revenge after Israeli air and artillery strikes on the coastal enclave killed a number of its members in recent weeks.
Israel has lost tracts of farmland and forests to fires set by kites and helium balloons, laden with incendiary material and flown over from Gaza. Locals said that Israeli troops, raiding the assailant's village on Friday, wounded a man.