Are you running multi QTP test from QC? When running several Tests from QC, the Remote Agent software by default will not shut down QTP until all tests have been executed. If you have very large tests and/or code that does not dismiss objects and variables the object might not get dismissed which will be retained in memory after the test is completed. The best way to clear objects and variables from memory is to restart QTP between every run. Since restarting QTP every run can cause an increase to run times of test I recommend shut down QTP between every couple runs. This is done by going to the Remote Agent and changing the setting to close down between every x number of runs. By default this is set to unlimited (value 0) and I would recommend a small number like 2 or 3. This will increase testing times because the wait for QTP to start up between each test, But since VB scripts are not the best memory management restarting of QTP prevent on memory resources.

Also, recommend to look at code for a coding problem where objects are not getting release by scripting engine. There are ways of coding to help the engine release the object. The best way to release object so that memory can be clean up is setting objects to nothing after being used.