1. Check to make sure that you have the correct Java version installed. Check the UFT PAM document for supported versions.
2. Make sure it is the right bit version of Java (32 bit or 64 bit depending on OS)
3. Check the environment variables against the examples in the Java section of the Addins guide. Pay attention to any spaces that might exist in the path. If you have any, make sure they are enclosed in quotes. This is a common problem.
4. Try reinstalling the Java Addin or UFT if needed.

Most of the time you can fix this problem by uninstalling the Java runtime and then reinstalling it all over again.

To do this, go to the control panel and then Programs and Features. If you're running Windows 8.1 Update 1, just right-click on the Start button and click on Programs and Features. Locate the Java runtime, uninstall it and reboot your PC.

Next, download the latest Java Runtime from java.com, taking care to install the correct version for your PC. You will likely want the 32-bit version as the 64-bit option is specifically for 64-bit browsers. Most browsers are 32-bit on Windows, even on 64-bit editions of the OS.

Sometimes this error can be caused by not having enough system memory allocated to the Java Virtual Machine - or not enough system memory full stop.

You can also try increasing the amount of system memory allocated to the [Java virtual machine][1] by setting an environment variable.

To do this, go back to the Control Panel and click on System, then Advanced system settings. Now click the Environment Variables button to bring up a list of all those currently set.

Mark

[1]: http://net-informations.com/java/intro/jvm.htm