What is the best approach to follow when Exchange Connector starts creating a loop of incidents? Is deleting the email manually from mailbox the only solution for it?

Pretty much right there. 

1. Disable the exchange connector
2. remove any emails in the mailbox that are not needed.
3. You should be able to stop looping by using the criteria on your workflows.
4. Enable the exchange connector

Are you trying on Exchange Connector 3.1 ? Multiple tickets for same mail got fixed in it. If you are on old version, would recommend to move to 3.1 

Are you trying on Exchange Connector 3.1 ? Multiple tickets for same mail got fixed in it. If you are on old version, would recommend to move to 3.1 

• Proposed as answer by Abhishek Verma [MSFT]Microsoft employee 3 hours 26 minutes ago

Are you trying on Exchange Connector 3.1 ? Multiple tickets for same mail got fixed in it. If you are on old version, would recommend to move to 3.1 

• Proposed as answer by Abhishek Verma [MSFT]Microsoft employee Sunday, September 06, 2015 3:48 AM
• Marked as answer by Harpreet25 7 hours 31 minutes ago

Are you trying on Exchange Connector 3.1 ? Multiple tickets for same mail got fixed in it. If you are on old version, would recommend to move to 3.1 

• Proposed as answer by Abhishek Verma [MSFT]Microsoft employee Sunday, September 06, 2015 3:48 AM
• Marked as answer by Harpreet25 Sunday, September 06, 2015 11:42 PM

Are you trying on Exchange Connector 3.1 ? Multiple tickets for same mail got fixed in it. If you are on old version, would recommend to move to 3.1 

• Proposed as answer by Abhishek Verma [MSFT]Microsoft employee Sunday, September 06, 2015 3:48 AM
• Marked as answer by Harpreet25 Sunday, September 06, 2015 11:42 PM