LIPSYNC

by Chris Clawson and Liubomir Kovatchev, aka "L. K."

*** PLEASE NOTE ***

This project was done in 2001,

using Blender 2.22, and Python 2.5. It is no longer supported and will not work with Blender after version 2.23. I will leave it on the website for reference, but please don't get mad if it doesn't work with the amazing, newer versions of Blender!

A Python tool for Magpie Exposure Sheets

Import Phomeme data setup in Magpie, and apply it to facial morphs in your Blender drawing, as RVKs

First, get Magpie HERE

Now download the LIPSYNC Blender demo

Magpie is a shareware audio mapping tool, which allows you to produce an 'exposure' or 'dopesheet' file. This is a text file, which lists the frame and time code, for an audio file, which you load into the program. By selecting regions of the audio file, you assign the different sounds of speech, known as phomemes, and write the phomeme names into the exposure sheet, into the frames where they occur.

A finished Exposure Sheet Example is here

In Blender, you begin by producing a face or mouth mesh, complete with morph targets for each of the phomemes, you wish to support. Each Mesh key should be one of the facial shapes made, for each of the sounds of speech. Then you set the Blender file for relative vertex keys, and initialize the curves, for each target. So, for each 'Key 1', 'Key 2', etc., you would insert a key at frame #1, with a value of zero. More information on setting up morph targets, and RVKs is here.

Take a look at the example dopesheet, in the white box above. Frame #4 begins the sound "aaa" or "ahh". It continues until frame #7. In Blender,'Lipsync' will read this information, and write a RVK for "A" into the frames from #4 to #7. Key events for "S" will be written for frames #7 and 8.

In this way, you may rapidly apply syncronized speech, to your Blender Model !

Alt-p the text window, in your example, and here is what you see...

1. DopeSheet File name: enter here the path and name of your Magpie exposure sheet file

2. Ipo name: select the correct IPO Block, in your drawing, which contains your mesh keys.

3. Save Setup to: enter here the path and name of the file you will create to save your setups. Entering or toggling this button will write your setups to disk.

4. (Phomeme Name): Place in these fields the exact name (and case) of the phomemes, as they appear in the exposure sheet. The first 3 columns only will allow single letter names, the last column will allow up to each characters each.

5. (Key Name): Select the key curve name that you wish to assign to the phomeme on the left. This will be the curve written to, any time the phomeme name (to the left) is found in the exposure sheet.

6. Import !: performs the import.

7. Auto/Vector Keys : A toggle button, which chooses the type of key handles to be written, either auto or vector. 'Auto' tends to produce smooth movement, while 'Vector' produces a more jerky, 'cartoony' movement.

8. Exit: exit the script. (make sure you have saved the setups, if you want them next time you start the script.

Here is what to do..

The demo file includes a dope sheet, 'list1.txt',which was produced from the example .wav file, included with the shareware release of Magpie. The words, "Hasta la vista, baby"!, are mapped into the dope sheet, and ready to import. The setup file, 'save.txt', contains all the correct mapping for this example. Unzip the Blender demo, and place the files into a new directory on your computer.

1. Open the file, in Blender, and change line 23, of the script, from "/project/lip/save.txt" , to your own directory, i.e. something like, "/yourdirectory/save.txt" .

2. Start the script, with Alt-p with the cursor in the text window. The menu should appear with the proper key mappings. If all the key assignments are "Key 1", then you haven't set the filepath correctly.

3. Press "Import !", and the script will produce the curves, according to the data in "list1.txt".

That's it !

Let me know if something isn't explained, or something doesn't work.

My thanks to Liubomir Kovatchev for the original idea, and continued work, testing and coding in this project.

I may not yet have learned to be an artist, but someday I might come up with a good idea.

Email Me

Now go listen to some strange old music