You should keep your ojects size always at real world scale if possible. Things like FG, GI, Caustics, AO … are related to it.
If you scale your object to big, rendertimes will be unnecessary long, if your object is too small, chances are that the settings go out of control to get a realistic/acceptable result. That is, smaller object means smaller values and there are no values below 0. And using 0.000… values aren’t the best choice.
So, scale does in fact matter and you can vary it. You just have to know what could happen and what caused it to be able to compensate it.
My suggestion would be, stay at real world scales at any time for as long as possible. It makes many things easier.