I encountered this problem on our SQL2012 and I have tried different scenarios (see below) to no avail. I have decided to give up and check if someone here has encountered this and resolved it. 

One thing I know, it's not a memory issue. Both servers we're using has lots of memory to spare and we monitor the memory as the replication goes through it's steps. 

I hope someone can help me on this. Thanks!

The Error:

The merge process could not allocate memory for an operation; your system may be running low on virtual memory. Restart the Merge Agent.

Our Scenario

• We're using SQL Server 2012 SP1. All subscriptions are pull based.
• We're using direct Merge Replication (not FTP or web sync)
• We already have 10 active replications with larger databases. Only 1 has this issue.
• Database size is less than 5 GB
• Rebuilding the publisher is not an option.

What have I tried?

• There is no memory problem --- we have lots to spare
• I have tried re-initialization of the database. Same problem.
• I tried deleting the database and reinitializing it. Same problem
• New snapshot. Same problem.
• I tried changing the subscriber server but still same issue. 

I know it is not a memory problem, but just out of curiosity - how much physical memory is on your server and what is sp_configure 'max memory' set to?

sp_configure 'max server memory (MB)'

Thanks for replying :)

The publishing server has 128GB of memory. The subscriber which is having a problem has 32GB.

I forgot to mention that we have other subscribers on this database and they are not having a problem. It's just this 1 particular subscriber.

If you are running a push subscription what does sp_configure 'max server memory (MB)' return on your publisher.

If you are running a pull subscription what does sp_configure 'max server memory (MB)' return on your subscriber.

Here is the result for the sp_configure on our subscriber. We're doing a pull on the server with the issue.

In addition, I made a comparison between the working servers and the one with the issue -- there seems to be a difference in the service pack. Publisher has none but the subscriber is operating on SP1. But still strange as only 1 database is affected.