Lip sync

Become a member of the CGSociety

Connect, Share, and Learn with our Large Growing CG Art Community. It's Free!

REPLY TO THREAD
 
Thread Tools Search this Thread Display Modes
Old 06 June 2017   #1
Lip sync

Hey folks,

I'm looking to do a character animation using some Daz figures that i will import into C4D as fbx, complete with their phenome morph targets, since I need to have the characters lip syncing to a speech audio track.

So, I've done lip syncing before in Cinema, but in that instance I was animating 2D planes, and switching out mouth shape graphics at the relative audio hit points, on a frame by frame basis!

I can obviously do similar with the morph phenomes, although I susoect I won't need to put in quite so many changes, due to the morphing between the shapes, right?

My main queries on this, are if anybody has any tips, or know if any way to speed this process up, or if in fact there are any useful plugins to help with interpreting the audio sounds to relative mouth shapes?

Cheers
Adam
__________________
www.kromekat.com

12 core Mac Pro / 2x Mac Pro 8-core ATI5770 / ATI4870
MacBookPro i7 2.66Ghz

 
Old 06 June 2017   #2
Originally Posted by kromekat: Hey folks,

I'm looking to do a character animation using some Daz figures that i will import into C4D as fbx, complete with their phenome morph targets, since I need to have the characters lip syncing to a speech audio track.

So, I've done lip syncing before in Cinema, but in that instance I was animating 2D planes, and switching out mouth shape graphics at the relative audio hit points, on a frame by frame basis!

I can obviously do similar with the morph phenomes, although I susoect I won't need to put in quite so many changes, due to the morphing between the shapes, right?

My main queries on this, are if anybody has any tips, or know if any way to speed this process up, or if in fact there are any useful plugins to help with interpreting the audio sounds to relative mouth shapes?

Cheers
Adam



Greetings good Sir,
You may remember me from the user name
"wolf359" from the Renderosity/Daz community
Good to "read"... you its been a while

But on to the questions at hand
I assume you will exporting from Daz Studio
as FBX and using one of the Genesis Rigs
as the old interposer pro/poser option has sadly become quite
obsolete IMHO.

I dont not know how much you are invested in Daz studio
but you may wish to consider buying the aniMate 2 plugin
which would enable MDD Export.

With an .obj/MDD pipeline you would get the animated mesh
in C4D with Lipsinc having been already created in Daz studio
via the built in Mimic plugin(32bit version only)
or the optional mimic Live plugin (64 bit but $50 USD)

I am 40 minutes into a feature length animated
non-commercial fan film
using a pipeline of :
1)Animate in Iclone Pro or using Daz non linear motion mixer graphMate spline editor
and keymate dope sheet plugin

2) Retarget to Daz Genesis one click( any Iclone motion Data)

3) Add Lipsinc from imported Audio via Mimic live

4)Export Static mesh from Daz to .obj with texture collection option

5) import .obj into C4D with texturesVia Riptide pro

6) Apply MDD Exported from Daz ,Light and render in C4D

Seems a bit "involved" I Know but its advantages for me, are
I can get a full HD resolution Daz mesh which may not be the
case with FBX
You get near realtime viewport performance in Daz studio
with single actors at base resolution but you export atany higher subD
you may prefer.

And any changes to the animation only requires export of a new
MDD file from Daz to the empty mesh actor waiting in C4D.

Here is a short clip from My Fanfilm Project All MDD animated meshes
animated/lipsynced in Daz studio, rendered in C4D.

https://vimeo.com/197307550

Apologies if this is not the type of suggestions you were seeking
But just thought I would share my
experiences/solutions for using the current Daz figures in C4D

You also may want to have a go at this plugin for an all C4D workflow
https://vimeo.com/99808070

The video is 3 years old but the DL link to the plugin seems
live as I am test downloading it as I write this
I know nothing about it beyond what is shown in the video.



Cheers

Last edited by ThreeDDude : 06 June 2017 at 10:00 PM. Reason: spelling
 
Old 06 June 2017   #3
I see 3 options :

- automatic : Daz studio (the 32bit version) has a lipsync editor (not the 64 bit I think). If you export any character as FBX with the animated blendshapes, you can then wire an expresso in C4D to transfer the animation data to your own morph targets. It's not really great imho, but it's there.

- capture : there are plugins and external apps to do face mocap. Among the cheap ones, you have brekel pro face which works with a kinect (1 or 2), or the newer Facerig Studio (it looks promising, not tried yet, available on steam).

- animate : you can setup an expresso of your morph shapes with a couple of 2D vector user data to blend between morph shapes (open/clone on one axis, stretch/pout on the other for instance).
 
Old 06 June 2017   #4
Quote: "- automatic : Daz studio (the 32bit version)
has a lipsync editor (not the 64 bit I think).
If you export any character as FBX with the
animated blendshapes, you can then wire an
expresso in C4D to transfer the animation data
to your own morph targets.
It's not really great imho, but it's there."


There is a 64 bit Paid version called Mimic Live
I am using it in my current project.
But Alas I see Adam is Mac based
and the Mimic live plugin from Daz is windows
only.
so indeed he may have to use the 32 bit Daz studio
if he wants an automated solution from Daz studio.
 
Old 06 June 2017   #5
Hey Wolf,

Long time no see! - thanks for your tips and an explanation of your workflow!

It's amazing how out of touch you can become, when you don't do one particular avenue of work for a few years. I gave up on Poser years ago, and was very happy with InterPoser for the following years, but I hardly ever do any work with these kind of characters these days.

2 years ago, I completed this animation;

http://kromekat.net/portfolio/gep-f...mation-project/

by using a Genesis figure in Daz Studio, and imported the fbx complete with all its pose morph shapes into Cinema, where I added some IK to simplify the animating. Thankfully in this case, I didn't need to do any lip sync.

I'll have a look at the various mimic options, since that could be the easiest method, although the DIM only gave me the 64bit version of Studio as an installable option, so I may need to request the 32bit version also.

Thanks to you both anyway, I may be back for some other specific questions within this workflow!

Adam
__________________
www.kromekat.com

12 core Mac Pro / 2x Mac Pro 8-core ATI5770 / ATI4870
MacBookPro i7 2.66Ghz

 
Old 06 June 2017   #6
Quote: "2 years ago, I completed this animation;

http://kromekat.net/portfolio/gep-f...mation-project/

by using a Genesis figure in Daz Studio, and imported the
fbx complete with all its pose morph shapes into Cinema,
where I added some IK to simplify the animating.
Thankfully in this case, I didn't need to do any lip sync."






Great job on that animation!! looks like something
one might see in the opening of a Disney movie.

If you go with the 32 bit mimic that ships with Daz studio
(32 bit) make sure you get the proper DMC file for the specific
figure you plan on using, it appears to make a difference in the quality
of the lipsync.

If it is the Genesis 3 female some reported having a hard time
tracking down her DMC config file.
I have a copy and I dont believe
there are any restrictions on my sharing DMC's
if you find you need one for her.

Best wishes with the project.
 
Old 07 July 2017   #7
Hey guys,

I've finally got the Genesis 3 assets in my DS, and just exported an FBX over to Cinema. I wasn't expecting each clothing item to have its own joint rig and be skinned separately - fairly sure that didn't happen previously with the original Genesis exports I did!?

Whats the best way to get these clothing items onto the main joint rig/figure in Cinema, and conforming to the skin weights of the base rig?

Adam
 
Old 07 July 2017   #8
Hi,

Is it not that you take care of this in DS?
When you use original DAZ Genesis clothing items you get one main rig in the FBX for all.
Did you make your own clothing or add older items?
In that case you should use the transfer utitity in DS to convert the new items before you export to FBX,

I'm no expert in this but recently did some DAZ fbx exports and also some C4D produced clothing items so that is my experience on this.

Cheers, odo
 
Old 07 July 2017   #9
Hi odo,

Thanks for your reply!

Yes its all Genesis 3 figures and clothing, I'm just using it with the Toon Generations 2 morphs - all works as expected in DS.

The last time I exported a Daz asset as fox, it, and it's clothing/hair etc all came in on the one joint rig, and were subsequently skinned as such. This time, each item has it's own copy of the same joint rig, and is skinned to that, but they exists as separate times.

Given that they are all based on the same joint rig, surely it's possible to retain the correct wait tag on, each, but have them be in the same rig?

If there is something I am missing in DS in that regard, cold you direct me to it by any chance? *or is there some workaround in cinema to transfer/merger themsot they all move in a unified manner?

Cheers
Adam
 
Old 07 July 2017   #10





Hi Adam,

I post a few screenshots of the workflow I used. Maybe that'll help?

odo
 
Old 07 July 2017   #11
Doh!

Now I see the export dialogue, I see the obvious! - thanks mate, I'll try again!
 
Old 07 July 2017   #12
Thanks dude!

I'm just trying the 32 bit DS and Mimic Lite plugin out. If you happen to have the DMC files for Genesis male/female, any chance you could direct me to them or share?

Cheers
Originally Posted by Adam!

ThreeDDude:
Great job on that animation!! looks like something
one might see in the opening of a Disney movie.

If you go with the 32 bit mimic that ships with Daz studio
(32 bit) make sure you get the proper DMC file for the specific
figure you plan on using, it appears to make a difference in the quality
of the lipsync.

If it is the Genesis 3 female some reported having a hard time
tracking down her DMC config file.
I have a copy and I dont believe
there are any restrictions on my sharing DMC's
if you find you need one for her.

Best wishes with the project.
 
Old 08 August 2017   #13
Hey guys,

I finally got to a point this week, where after building assets for the project I am working on, that I could add lip sync to a character and export to Cinema 4D for further animation.

This all work ed well, with a 1 minute voiceover audio file, applied to the Genesis 3 figure, utilising the correct dmc file, and assisted by a .txt transcript. I then had to save the file in the 32bit Daz Studio, and reload in 64bit, to do an export with both animation and morphs attached, as it was too much for the 32bit, which crashed on export.

All fine and good, and working great in C4D!

However, I now have a 10 minute audio file I need to apply to another Gen 3 figure, and when I try to analyse it, it errors and stops fairly quickly. This I assume, will be a memory limitation of the Mimic plugin in the 32 bit DS!?

So my only other option, appears to be to buy Mimic Live, and install on an old PC laptop we have here. Before I do that though, can anybody who has experience with Mimic Live tell me if it has any memory limitations on audio files?

I'd do it in smaller sections, but in the existing Mimic I have, only one audio file can be loaded and processed on the character, and I'd need to keep exporting separate versions of the same figure, which all need to be setup again in C4D, and then made into one seamless animation - ball ache!

Any advise, or knowledge on the capabilities of Mimc Live, would be great!

Adam
__________________
www.kromekat.com

12 core Mac Pro / 2x Mac Pro 8-core ATI5770 / ATI4870
MacBookPro i7 2.66Ghz

 
Old 08 August 2017   #14
I've worked around the problem,by slicing the audio into 3 parts, which the 32bit mimic could handle, and have imported each version of the character into C4D. I'll simply have to hide and show each of them in sequence to achieve a seamless, single speaking character.
__________________
www.kromekat.com

12 core Mac Pro / 2x Mac Pro 8-core ATI5770 / ATI4870
MacBookPro i7 2.66Ghz

 
Old 08 August 2017   #15
Originally Posted by Kromekat: Any advise, or knowledge on the capabilities of Mimc Live, would be great!



Hi Adam, Glad you found a workaround
I have 64 bit Mimic Live as well as the older stand alone
Mimic 3 pro suite.
I use both, depending on the situation.

Just a word of caution if you plan to get the 64 bit Mimic live
for future projects.

It literally creates the lipsynch animation in realtime from "live Audio"
and has no option to load a pre existing audio file.
However ,as usual, there is a workaround that I have found is quite effective
I explain the process in this linked Daz forum thread:
https://www.daz3d.com/forums/discus...mic-live#latest
 
reply share thread



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
CGSociety
Society of Digital Artists
www.cgsociety.org

Powered by vBulletin
Copyright 2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump
Miscellaneous

All times are GMT. The time now is 12:36 AM.


Powered by vBulletin
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.