Recommend trying UFT 12.01.

Make sure you have enough memory to run UFT and Application Under Test (AUT). UFT Readme states:
Minimum of 2 GB when no more than three add-ins are loaded simultaneously
Note:

- An additional 512 MB of RAM are required when using a virtual machine.
- Additional memory is required when loading more add-ins and when using the Save movie to results option to capture movies during run sessions.

How big is your test? Do you try to test the complete AUT in one test? Recommend breaking test into smaller test.

Do you dismiss your object variables after using them?

Recommend reviewing HP Document: [KM197527][1] Title: ''Virtual Memory errors occur during long test runs''

[1]: http://support.openview.hp.com/selfsolve/document/KM197527