I have an existing exchange 2010 SP3 organization.  I am planning to install Exchange 2013 RTM on a production server
to try it out. Can the RTM version coexist with exchange

In coexistence scenario you must install at least Exchange 2013 CU2. RTM and CU1 versions are not supported.

Yes , it is supportive . it is better to go with  Exchange 2013 CU3 .

http://technet.microsoft.com/en-us/library/jj898583(v=exchg.150).aspx

Hi,

Lets begin with the following article:
http://exchangeserverpro.com/installing-exchange-server-2010-service-pack-3/
The co-existence support for Exchange 2013 does not apply to Exchange 2013 RTM, but rather Exchange 2013 CU1 (cumulative update 1) due for release in Q1 of 2013 (within about 6 weeks from the time of this writing.

I recommend you install the latest Exchange 2013 CU3 version.
And here is a reference about how to install Exchange 2013 in an existing Exchange 2010 organization:
http://technet.microsoft.com/en-us/library/bb124350(v=exchg.150).aspx

If you have any question, please feel free to let me know.
Thanks,
Angela