The alleged offence on a 38-year-old woman took place in the grounds of St Mary and St Eanswythe Church, Priory Gardens, at about midnight on Thursday.
A 30-year-old man from Canterbury and a 31-year-old man from Folkestone attended Folkestone police station on Saturday and have been released on bail pending further investigations until November 14.
A third man, aged 29 from Folkestone, is currently being questioned in custody after attending Folkestone police station earlier today (Monday).
Police are also trying to trace a fourth man in connection with the assault.
Acting Det Chief Insp Julien Lawton said: “The victim had visited two local pubs that evening – the Lord Morris in Guildhall Street and the East Kent Arms in Sandgate Road.
“While she was in the East Kent Arms she had a conversation with a group of four men. They were all white, aged about 28 and were all of slim build and dressed in casual clothing. One man had tattoos and a foreign accent.
“She left the East Kent Arms at about 11pm in the company of one of the men. It appears that he was then joined by three other men, and they took her into the graveyard where she was sexually assaulted.”
Police have searched the graveyard and are also carrying out house-to-house inquiries and examining CCTV footage of the area.
Anyone with information should call police on 01622 690690 or Kent CrimeStoppers, anonymously, on 0800 555111 quoting crime number FY/015285/11.