Suggest call Tech support and have a look at the content of the webtrends\logs\install and webtrends\logs\uninstall folders for errors in the installation log files created.
I've also had problems upgrading to 8.7. The first time it was my fault trying to go from 8.5 straight to 8.7 instead of 8.5a. After the install backed out i came to find all the services were gone now for webtrends and when i tried to run 8.5a install or even the 8.7 it couldnt determine the current version. So i backed up to 8.0 and started from there and now im up to 8.5a. I made sure to backup my db's and config before starting any of this of course. So hopefully this works this time. I have already had one error though.
ERROR: Unable to open glue library configuration file
I had this exact same issue and when it rolled back it deleted all traces of WebTrends on our system.
WebTrends support suggested the issue is related to lack of resources but, our server has 4 processors and 8 GBs of RAM I sure hope that is enough resources to perform an application upgrade.
They also suggested we uninstall the UI from the Application Server and then move the UI to another server, then try the upgrade. My problem with this is why if they clearly state in all their documentation that a single server can support all services should I have to remove the most important service to perform the install. If this is truely the case and I have not had a chance to test it yet it would clearly point to a bug in their upgrade process that needs to be resolved IMMEDIATLY. Any upgrade that is causing muliple organization to loss all of their data is a CRITICAL bug.
The update from 8.5a to 8.7 also failed on our system. The support team of our WT distributor told me to do a modify setup and uninstall the UI server from the failed installation, then rename some folders and add the UI back in. The result was that all WT services were deleted from the system, the setup routine does not run anymore, and the system remains non-functional.
I requested to our distributor to have someone from WebTrends call me and repair this in a LiveMeeting session. Curious if they can fix it.