People are being urged to stay away from seals that are hanging around a south western Victorian coastal town breakwater boat ramp.