There's a couple of things that could cause this.
QTP 11 does not support usage of seat licenses under Windows Server 2003 and
Windows Server 2008. Make sure you have Concurrent License checked and set up properly. You can check/change this by going to Start>All Programs>HP QuickTest Professional>Tools>Additional Installation Requirements and re-run the License Installation Wizard.
Somtimes people accidentally install QTP Essentials instead of the full QTP package. This causes problems too. You can convert a QTP Essentials install to a FULL QTP install by launching the registry editor and changing the following key:
HKEY_LOCAL_MACHINESOFTWAREMercury Interactive/Quicktestprofessional/Currentversion/
Change to
HKEY_LOCAL_MACHINESOFTWAREMercury Interactive/Quicktestprofessional/Currentversion_BKP
Also, look for keys ''QTPEssentialsPatchInstalled'' and ''QTPEssentials''. If they exist remove them from registry and reboot then launch QTP again.
I have encountered this situation where the registry key did not exist. In this case, a full/clean uninstall of QTP Essentials and a re-install of the full QTP version is required.