yeah under vaccum it will suck air from the intercooler hole, into your engine (without filtering) under boost this will leak some of the air your turbo worked so hard to compress and heat up, and you will notice maybe a small drop in performance and boost levels.
EDIT: yeah you can probably fix the leak, as the intercooler is aluminum, or you can get a bigger intercooler. do a custom front mount. people have found that intercooler is good to around 15psi. check ebay they have some decent intercoolers for a decent price. you can probably have a FMIC for under 200 bucks with new pipes (EBAY) or you can get another stock intercooler (or 2) and have someone (unless you can weld aluminum) make you a dual or tripple core intercooler.