Generally it is not recommended, but what you are asking about is what is called an ''Upgrade in place''.
When you install QC/ALM11.52, ALM11 must be uninstalled first (but you can leave the DB's and repository folders where they are).
This can be dangerous as sometimes the upgrade process can take longer than expected - thereby keeping users out of projects. Also, you have no ''safety net'' to fall back on should you need to quickly restore or re-activate a project in the ''old'' QC11 server.

If you must do this upgrade in place PLEASE make sure to MAKE BACKUPS of ALL QC-related DB's and corresponding repositories (file system).

Make good notes about the fact that these go together and were for QC11.

Find out the ''confidential data and security passphrase(s)'' and write them down -- they will need to be EXACTLY the same on the NEW load of 11.52 as they were for 11.

If nothing else, I recommend loading 11.52 (and patching to Patch 1 or Patch 2) on a ''test/dry run'' box, then copying project DB's to a separate (new) DB server and repositories to the new server, then do the usual RESTORE / VERIFY / REPAIR / UPGRADE process.
This will allow you to discover any upgrade issues in a ''safe'' environment (copies of project DBs and repositories) and to take metrics of how long various parts take.

You will likely want to upgrade the qcsiteadmin_db database (top level QC information stored here), but you can create a NEW one as you load ALM11.52.

Also, QC/ALM11.52 Server REQUIRES 8GB RAM, that is easiest to accomplish on Win2008x64 server or Win2012x64 server (assuming Windows).