A Subwoofer is a piston it moves air back and forth. With out the right enclosure your sub will not function properly.
Even ported boxes are dampening the movement of a subwoofer to make the right sound.
A Woofer that calls for a bigger box than you put it in will have a stiffer surround. Putting it in too small of a box will make it over dampened, resulting in reduced output and low frequency extension. You can add material to to large of a box to help but is best to get as close as you can to its recommended volume.
A woofer that is designed for a small box will have a loose surround. using a woofer in too small of a box will produce boomy bass (muddy or not clear and crisp) and can be easily damaged.