Processing of 3-D medical images improved

[Crocodilian]

Computer scientits at the Mayo clinic have come up with new ways to process faster 3-D medical images.

>>LINK<< (http://www.sciencedaily.com/upi/index.php?feed=Science&article=UPI-1-20070410-09145000-bc-us-images.xml)

-R

Followed the link, which isn't terribly informative, but does suggest what this is about, and its not all that useful to CG work. Medical "imaging" is only distantly related to CG work that folks here do. . . the story in question involves the problem of "sensor fusion". A more detailed article is at
http://www.advancedimagingpro.com/web/online/Industry-News/Boffins-Makes-Breakthrough-in-Real-time-3-D-Medical-Imaging/3$3870

Basically, let's say I have an MRI and PET scan of your brain-- and I want to overlay them to obtain different kind of data for the same point in 3space so that I can evaluate a given part of the brain in both spectra. Or, even more complex, let's say I want to overlay electophysiology data (electrical potentials) over the myocardium of a beating heart.

These are hard "registration" problems, where surfaces are deforming and where data has been acquired with differing time scales, spatial resolutions, and so on. Its hard to think of anything that we do in the CG industry that might be analogous-- certain aspects of plate matching/compositing come close, but these are seldom the true volumetric matches that are required in medicine.

Having spent a portion of my career working with people doing medical imaging, I was suprised how little overlap there is between their techniques and those of the CG world; they "see" the world very differently . . . and as yet, there's been little crossover between disciplines.

[cresshead]

on a related note..the lightwave plugin for such things as 3d medical scan data
http://www.mediastudio-graz.com/volumedic//html/vm_gallery.htm

and a sample video
http://www.mediastudio-graz.com/volumedic//objects/vm_ct_anim001.mpg

[E_Moelzer]

Thanks for the plug Stephen.
:)
I had the chance to work with "Analyze" by the Mayo clinic in the past.
Very capable piece of software and very capable people working there.
Its always cool to see advancements in this area being made.
Analyze influenced the development of our VoluMedic in many ways.

CU
Elmar

[Mic_Ma]

[...] I was suprised how little overlap there is between their techniques and those of the CG world; they "see" the world very differently . . . and as yet, there's been little crossover between disciplines.

Do you think there could be convergence in the future? It can't be exclusively polys and surfaces for the next decades, me thinks. I'd like to see some melting of technologies.

[Crocodilian]

Do you think there could be convergence in the future? It can't be exclusively polys and surfaces for the next decades, me thinks. I'd like to see some melting of technologies.

Its a great question. I don't think there will be convergence-- because the objectives of medical and entertainment disciplines are so different. From "our" perspective, we think of surfaces . . . volumetric calculations are just too painfully slow.

Take an application like Wondertouch's Particle Illusion (also known as the particle sytem in Combustion). Its an entirely "fake" volumetric application, but for my purposes, its infinitely preferable to a true volumetric solution.

Medical would have no use for a sprite-based "pseudo-volumetric" calculation . . . they deal in "true" voxels, and only in "true" data, for obvious reasons. "We" have little use for their exacting volume data, as we can cheaply "fake" what they have to expensively simulate. Take skin, for example-- we do skin with a surface, and a SSS algorithm, that "fakes" the way light moves through volumes (which we don't noeed to define). When medical does radiation treatment planning, they can't "fake" anything, least of all the behavior of beams as they pass through tissue (though x-rays have very little scattering in tissue)

I think there will be crossover as time goes on-- entertainment spends way more money, and as an entrepreneur I've tried to get medical imaging guys off oddball special purpose hardware and onto standard hardware, with mixed success.

[Mic_Ma]

. . . volumetric calculations are just too painfully slow.


Yes, currently the hardware is optimized to handle pixels and vertices through dedicated shading languages. The mathematics and algorithms to handle these things is well developed and fairly ubiquitous. Software tools have evolved to enable fast and efficient workflows...

But I hope that one day voxels will come back. I feel that there is no intrinsic reason for them to remain in the shadow of the mighty polygon. I am sure that sooner or later some bright sparks will work out ways to streamline the whole thing and all kinds of wonderful benefits for the artists will materialize.

I always find it odd that we ended up to with 2 dimensional surfaces to represent things that are effectively volumes.

Perhaps market forces will eventually spur on developers to poke new technologies into our traditional toolsets.

[Crocodilian]

But I hope that one day voxels will come back. I feel that there is no intrinsic reason for them to remain in the shadow of the mighty polygon. I am sure that sooner or later some bright sparks will work out ways to streamline the whole thing and all kinds of wonderful benefits for the artists will materialize.

I always find it odd that we ended up to with 2 dimensional surfaces to represent things that are effectively volumes.

One answer is because "we don't have volume data". Medical now produces a huge amount of volume data: PET, CT, and MRI scan data are volume data, and resolutions are breathtaking.

Another point: Its hard enough to model surfaces-- how would one model volumes? I've played around with Lightwaves volumetric materials-- which can be driven by procedural textures . . . fun, but desperately slow.

There are real innovations in software -- think of ZBrush, for instance . . . but artists need interactivity, while medicine needs accuracy.

Another key difference: artists look at the visual specta, where light interacts in a most complex manner with organic materials (that's probably why we see the wavelengths we do-- because they're most informative). Medical imaging occurs in parts of the EM spectrum which behave very differently than visibile light.

[DrBalthar]

The reason why we have surfaces is simple because we human see mostly surfaces we do not have x-ray vision so there is no need to see the whole volume. There are a few exceptions like some fluids and gaseous phenomena.

And even in medical visualization I have the feelingthat it seems that the most useful thing is to extract a bunch of iso surfaces than to look at the volume mess that MIP,DVR produces. (At least from my point of view I just see blurry images in that but then again I am not an expert)

[Mic_Ma]

Another point: Its hard enough to model surfaces-- how would one model volumes? I've played around with Lightwaves volumetric materials-- which can be driven by procedural textures . . . fun, but desperately slow.

Another key difference: artists look at the visual specta, where light interacts in a most complex manner with organic materials (that's probably why we see the wavelengths we do-- because they're most informative). Medical imaging occurs in parts of the EM spectrum which behave very differently than visibile light.

I have have experimented with a haptic modelling device a few years back at university. I don't recall the name of the software (not a common one, perhaps custom?) . The model appeared to be made of voxels. Objects could be manipulated very much like clay.

I am speculating here but I think that voxel-clay may be easier to handle mathematically than a poly-clay. Pushing, folding, ripping and reglueing such poly-clay would surely cause terribly topology problems. Voxels could have a much simpler atomic behaviour, each one being pretty much topologically independant from its neighbours. With such a truly volumic material you could add plasticity, viscosity and other mechanical properties into your modelling.

These are half baked ideas, but what I am saying is that there may be opportunities in a voxel approach, and that we shouldn't be myopic about it merely because no solution is staring us into the face. 30 years ago, who would have thought about all the amazing poly magic we do nowadays.


I think spectral rendering can deal with x-rays and such things. Light is part of well understood electromagnetic radiation.


Come on future!

[Crocodilian]

The reason why we have surfaces is simple because we human see mostly surfaces we do not have x-ray vision so there is no need to see the whole volume. There are a few exceptions like some fluids and gaseous phenomena.


Its fascinating to consider the evolution of sensory organs. We take hearing and vision as a given, they're not, obviously. The visual wavelengths a very narrow band, which has the unusual characteristic that it is diffracted by organic tissues. Wavelengths which are either longer or shorter don't scatter in the same way through organic materials.

One of the effects of this-- which we see everday in real life, and now also everday in CG-- is radiosity. In daytime, our hole world is lit by the sun -- just incredible, when you think about it. . . neither an IR source nor a UV source, not to mention an X-ray source, would behave this way.

So while critters see in slightly different segments of the spectrum, I would suggest that our senses are exquisitely attuned to our evolution. Note the brilliant apparent contrast between red and green; that's not a "given" rather its how we evolved-- great for picking fruit out of trees.



And even in medical visualization I have the feelingthat it seems that the most useful thing is to extract a bunch of iso surfaces than to look at the volume mess that MIP,DVR produces. (At least from my point of view I just see blurry images in that but then again I am not an expert)

This is done a lot in the visualization side --- when you're trying to make nice images-- but there's an interesting distinction. You'll hear medical imaging folks refer to "imaging science". We CG types think that they're talking about the same thing we're talking about (sometimes, they do too), but they're not. So the distinction between "medical imaging" and "medical visualization" which seems very obscure, actually exists and is important.

Most "Imaging Scientists" are working with volume data, and are uninterested in surfaces, except when they have to produce pictures for their journal articles. When trying to do surgical treatment planning (probably the leading medical imaging science application), they're trying to identify anatomical units and their volumes, so that a surgical approach, or radiation beam can be calculated. They think in voxel terms, because those are the data that come out of PET/MRI/CT. The only medical sensing technology I can think of that returns values on a surface is an EKG (electrical potential over the myocardium)

[E_Moelzer]

Well, without wanting to steal the thread, but volumetrics can render pretty fast these days (and the renderings can look just as good, if not better than polygon renderings). Not quite as fast as polygons, but the gap is closing. VoluMedic can render full Pal res images with raytraced shadows in seconds.
The bigger problem is the amount of data. True volumetric objects require a lot of memory because you really have an "inside" for everything. So what you do is reduce the amount of data and then interpolate it, which again costs processing time.
The other problem is in the animation. Deforming volumetric data is not that trivial (again a lot of data needs to be deformed and processed) and then it often does not look good. There is still a lot of work that needs to be done in this area (and it is being done).
Getting volumetric data is not that big an issue though, IMHO. There are many sources for medical data all over the web. Our (soon to be released) VoluMedic 1.5 also offers volume painting which allows to paint into the volume with a volumetric brush, to generate new shapes or modify existing data.
This is actually very intuitive and easy to use.
In addition to this, we offer tools that allow to convert polygon objects into volumetric objects.
If volumetrics had as much backing in the industry as polygon modeling has had for the past decade, we would certainly see a lot more excellent "modeling" tools for them and I am sure that they could be replacing polygons in most areas.
The question whether this would be really reasonable to do, is of course a different one. You dont always have to see the inner workings of something.
For areas like medical imaging, material sciences, certain earth sciences and certain visual effects(like fire smoke, fluids, clouds, etc), volumetrics are certainly the best solution though, IMHO.
Oh and VoluMedic does have tools to convert the volumetric data into polygons.
CU
Elmar

[Crocodilian]

No, not you're not thread-stealing! Seems to me that the original article lead to a range of comments/inquiry about the relationship between medical and fx-style imaging.

Your comment about data is well taken-- and I'd extend it slightly. Medical imaging produces rich volumetric data sets, but these data are only encountered in a few other places. Meteorology is probably the place where volumetric data is most commonly massaged into entertainment-grade visualization (all those pretty pictures on your nightly newscast weather report).

The FX world generally doesn't capture or produce volume data (eg, we've yet to attend a volumetric movie).

Processing of volume data in medical imaging is tricky; what comes out of the scanner isn't automatically useful for anything. The hard part is "segmentation", distinguishing tissues from each other. This has historically been done manually, with grad students working on a tablet and going slice by slice through scan data, circling and labeling structures. This is tedious and slow, so there's been a lot of work on automating this process-- but its a very hard problem.

[E_Moelzer]

I did a lot of segmentation in my old days when I did medical viz at the local hospital.
Its not as bad as you think. E.g Analyze offered region growing back then, which pretty much automatically went through a marked region in for every slice based on a single input by a user. Every 20 slices or so, you might have had to correct some things, but thats not more difficult than using a magic wand tool in Photoshop.

In VoluMedic, we have multiple ways of doing segmentation (and there will be a few more in future updates):
1. By luminance value (in case of a CT-scan this would be the optical density in the xray spectrum). This is very quick and easy to use. You can segment bones from tissue (or different kinds of tissue) within seconds that way.
2. By using so called "Bounding Boxes". Just stack as many of them on top of each other and use those to get rid of the parts you dont want (the boy name is kinda wrong since 1.5 since they are now also spheres and cylinders).
3. By using geometric shapes to remove parts from the dataset.There is a rendermode that allows rendering geometry into a dataset (the transparence of the surface on the geometry determines the density value of voxels that are inside the geometry).
4. By using Volume Painting to paint away parts that you dont want.
5. By combining any or all of the above.

Now while with each of the methods above by itself segmentaion of individual organs might still be difficult, it is getting much easier the more of the others you use in addition to that.
All these are rather easy tasks and usually much easier to do than modeling e.g. organs in a modeling application by hand and then they are very precise and actually based on real data from a real person.
CU
Elmar

[Crocodilian]

Hi Elmar,

fascinating stuff. Volumedic is very impressive -- I had not seen it before. You are doing exactly what I had hoped would happen when I was working in this field about seven years ago-- using robust 3D applications to interpret scientific/medical data. This is a creative and useful application.

My work was with academics who developed automated segmentation algorithms, fairly complex stuff from pattern theory, deformable templates usually. The leading paper in the field was by Grenander and Miller "Computational Anatomy: An Emerging Discipline".

Johns Hopkins University has a Center for Imaging Science which does a lot of work in the field:
http://www.cis.jhu.edu/computationalanatomy/computationalfunctionalanatomyoverview.html

My experience in working with folks in this area was that the research domain was surprisingly distant from what CG professionals are able to use . . . and there was very little communicaton between the imaging science researchers and CG types. . .

[E_Moelzer]

Yeah, I noticed the same thing back then.
I did a lot of medical viz at the hospital. All using expensive specialized software. What I noticed back then was that these applications were extremely "clumsy" to use. They felt more like experimental applications or something that was work in progress, or like a custom inhouse tool some game developers use, especially since I had a direct comparison to LightWave, 3DS and Softimage back then.
Thats why we got the idea to develop VoluMedic in the first place.
Unfortunately it took a loong time to get it released, simply because we were lacking manpower and money. Anyway, VoluMedic is out now and it is the tool, I wished I had back then, when I was doing medical viz.
CU
Elmar

[CGTalk Moderation]

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.