Japan said the statue violated the spirit of a 2015 deal meant to settle the decades-long issue with a Japanese apology and payment of money to survivors.
The minister also said that Nagamine is expected to hold talks with Hwang Kyo Ahn, acting president of South Korea, after his return to Seoul and urge the country to implement an agreement reached in December 2015 between the Japanese and South Korean governments to "finally and irreversibly" resolve the issue of so-called comfort women.
That one, which has become a symbol for activists campaigning on behalf of the few surviving former sex slaves, still stands and Japan saw the new one in the southern port city of Busan as unacceptable.
A senior South Korean Foreign Ministry official on Monday expressed a wish that the ambassador's return to Seoul would promote bilateral communication.
The foreign minister noted the importance of both Tokyo and Seoul maintaining a high level of intelligence-sharing and close dialogue with regard to North Korea's missile and nuclear programs.
Speaking at a briefing at the Ministry of Foreign Affairs, Kishida said that the move to return Nagamine to his post was necessary ahead of a presidential election in South Korea on May 9 and in light of the fact that Japan needed to focus more efforts to prepare for the next administration.
Japanese Ambassador to South Korea Yasumasa Nagamine arrives in Tokyo on January 9, 2017.
Chief Cabinet Secretary Yoshihide Suga said Japan's other retaliatory actions, such as the suspension of some negotiations with South Korea, including a proposed currency swap arrangement for times of financial crisis, will stay in place.
Japan's consul general in Busan, Yasuhiro Morimoto, who was also recalled, will join Nagamine in returning to South Korea, Kishida said.
"The agreement is a promise that the two countries made to worldwide society".