For some reason Maya doesn't know that the attribute exists unless the script is done executing. After the script is done, there's no problem accessing the attr from mel but there's no way to do it from within the script.
It should be as simple as:
vray addAttributesFromGroup $shaderEngineHandle vray_roundedges 1;
setAttr ($shaderEngineHandle + ".vrayRoundEdgesRadius") $roundEdgesVal;
Error: setAttr: No object matches name: <SGname>.vrayRoundEdgesRadius
I've tried "refresh", "updateAE", "refreshAE"... what am I doing wrong?