First of all, ensure that this option is enabled in your profile. For that, click on the Profile link in your top navigation bar and go ahead to the "Preferences" section. Tick "Yes" for the entry "Pop up window on new Private message".
If it's already done, then your problem is caused by a popup blocker. Some navigators are equipped with a popup blocker. Below are some instructions on how to disable the popup blocker in your browser (IE and FF). If you another browser, check the helpfile or search within the options or settings.
Firefox:In your main menu, on top of the address bar, find the menu entry "Tools" and select "Options". Click on the "Content" tab. You will see the exceptions button now nearby the label "block pop-up windows". Click it and enter your forum URL as allowed site. If you have a customized domain name, you should add both the custom domain and your original forum address.
Internet Explorer, on Windows XP Service Pack 2/3:
Click on Tools (besides Favorites) then on Internet Options
Click on the Privacy tab
In the Pop-up blocker section, go to Settings
Type the forum address (you can omit the
http://...), then click on Add
Google Chrome (solution thanks to Loki)Options > Under the Bonnet > Content Settings > Pop-ups > Allow all sites to show pop-ups
If you have the MSN toolbarWhile you are on the forum page, click on the little arrow just beside the Blocked pop up, on the toolbar, then click on Allow this site to open pop up windows. The pop up windows of the forum will be then allowed.
Tutorial from: http://help.forumotion.com/frequently-asked-questions-f5/i-recieved-a-pm-but-no-pop-up-opens-t8331.htm