Here's what you need to understand about VMF and BSP files:
A VMF is a raw, uncompiled map file. It can be opened in Hammer and edited from there. A BSP is a compiled map. You can run it in whichever game it was compiled for. However, you cannot open BSP files from your file browser, which is why your computer will give you an error if you try to open one this way. To open a BSP file, you need to launch your game and use this console command:
map <file name>
Replace <file name> with the name of the BSP, and do not include brackets or the BSP file extension.