Well, there are tricks that can be done committing shading to lightmaps which people use to be able to display procedurals. You can have a look in that direction and search the forum a bit, but it tends to introduce an edge of instability to the process sometimes.
Ultimately what you talk about remains possible. Rendermapping or photogrammetry like techniques can be used to produce animated sequences that will drive something else exactly like you describe.
The problem isn’t really impossibility (you seem aware of this though), as much as the feedback loop to do it having this really annoying interval (rendering the maps) half-way through it.
Look into lightmaps though, maybe it will fit the bill for what you want. Can’t say much more since I’ve been rather unconcerned with anything MRay that isn’t volumetrics for a while now.