Woof, this was hard, but is done enough.

Talk about frustrations! This is what proofs of concept are all about, after all, but still, one would hope for somewhat easier going. I learned a lot. I think the output is also kind of neat, although the technical execution isn’t up to the standards I see in my mind’s eye. I’m sure the finished movie will have the same lament!

First, the result. Purpose: synchronize audio to the fractal. Done, April 16 2007 — see the thin orbit traps wriggle with the bass, and the lead part make a layer get more visible. Part of Study #5 in here as well: hand-keyframing large scale fractal parameters to the music. What a pain in the ass. Here it is:

Ok, here’s the nitty gritty for those fellow explorers in this bizarre corner of the creative universe.

Rendering

This is a one minute video, and due to various problems, took almost 3 elapsed weeks to render. Yikes, this is a huge problem. That’d lead to 6 years of rendering to get my film out…

First, all rendering jobs in Ultra Fractal get submitted by hand. This is fine for one-offs, but gets especially bad when you’re trying to work multiple layers with lots of tweaking. I asked Frederik Slijkerman about the possibility of controlling UF by the command line to better integrate it in to an animation rendering pipeline. Although he was very gracious about responding, and open to the ideas I had, how can I expect him to add features that are of interest to 1/10th% of his market? Not likely, especially in the timeframe I’m needing.

Overall, Study #7 (”Make it Efficient”) is really seeming important to me. It’s getting in the way of completing studies, much less the actual movie. I’m going to prioritize this as the next thing I need to solve.

Some of the problems I encountered were:

• Submitting jobs, by hand, with the wrong parameters (eg, dimensions of 1440 x 720 instead of 1280 x 720)
• Manually having to turn layers on and off to separate them so I could control them in final compositing instead of in UF
• Getting intermediate results for sequences that took 100 hours to render per layer — I had to manually split them up into smaller increments, with the very real opportunity for screwing up every time I submitted
• Relying on workstations that my family and I use during the day

Mistakes galore, which all require re-rendering, and not a small amount of unhappiness with all the family computers being sluggish because they were rendering.

I came up with an interesting solution for the wrong dimension problem. I just couldn’t face re-rendering 1000 frames, at about 70 hours. I instead brought both the first 400 frames with the right dimensions, and the rest of the wrong-sized frames into Shake. I set up a simple keyframe switch to select between the original frames and a cropped version of the later frames. Voila, solution in about 1 hour of dinking around and no re-rendering in UF. I’m really liking Shake as the swiss-army knife processor.

I will write more about Study #7 shortly — been giving it a *lot* of thought — but for now, the summary is:

• I’ve created a command line wrapper around UF to submit jobs. Brittle, but works
• I’m either going to adapt ruby-queue or write my own queue to manage distributed job queuing.
• I’m going to order four render farm machines that will be dedicated 24×7 to rendering UF frames
• I’m going to set up QMaster to spread out Shake and Quicktime compression jobs
• I’m going to set up better networked file shares for the render machines, and will devise strict naming conventions to keep things straight
• I’m going to get vicious about working small and fast (320×180) until my ideas are settled down. Rendering and manipulating HD is ridiculously slow.

Codecs

I was totally pissed when I brought this hard-won content into Final Cut and saw that it looked like roadkill. Weird artifacts, obvious pixelation, colors that were off… blech. I’m going to explore codecs in more detail later, because they are really key for quality. I’m pleased with the Final Cut Studio 2 announcement from yesterday: lots of terrific features, will be sure to order. However, I’m not sure the new codec it introduces really solves my problem; it appears more geared to HD from tape.

So I jumped ahead a bit in my explorations, trying to get a reasonable result. When I first started to put together sequences in Shake, I rendered to Animation, 30 fps, Millions+. Final Cut barfed on this. It wouldn’t display the clip in the viewer or canvas, and wouldn’t render it. I then tried Apple Intermediate Codec, 720p30 at 29.97 fps. Final Cut liked this a lot better, but it looked terrible when I finally rendered it. After a few fruitless hours of googling, I got the bright idea of changing the Shake encoding to Animation, 29.97fps, Millions color. Note the Millions, not Millions+. I didn’t have alpha in these particular clips, so no sacrifice and a significant decrease in file size and it actually worked in Final Cut!

At this point, I’m confused as to why this worked and the other settings didn’t. After all, don’t Motion and LiveType produce Animation with Millions+? Dunno, will require some tedious research, but at least I can produce acceptable results. However, workflow in Final Cut sucks; touch anything, and am forced to re-render. This takes 10 minutes for a 1 minute video, and this is on a quad-proc MacPro. Study #9 (”Codecs Kill”) is important to nail down. I will wait until I actually own Final Cut Pro; I’m suspicious that my consumer level Final Cut Express has limitations baked in that the Pro version doesn’t.

Synchronize Audio & Music

Mixed results. Love the potential, hate the process I went through. Essentially, had to hand-key everything. Total pain in the ass. There’s no way I’m doing this for an entire movie. Then again, that was somewhat expected, so no huge surprises.

First, Study #4 was all about getting Shake to do cool things with audio. This was a success, although it ultimately required hand-intervention that makes me question the real utility of it. I had one layer that I wanted to squiggle in time with the bass. I solo’d the bass track in Logic, and exported just it. I imported that into Shake and then used its audio-to-curve analysis to create a curve that I used to drive the x and y coordinates of a warp node. I played with the formulae for a bit (taking it mostly from the Shake manual) to come up with x = x + 6 * audio_L * sin( y / ( 6 * audio_L ) ) and y = y + 7 * audio_R * sin( x / ( 7 * audio_R ) ). This looked cool, but when I set the sampling of the audio to a small time, it vibrated too much; I had to set it to sample a lot less frequently. This looked good, but there was too much delay from when the sound started and when the squiggle started. Thus, I had to hand-tweak the animation curve. I think I could have created an equally good looking curve from hand by looking at the audio waveform and just drawing. So, not sure this is incredibly way cool hip or just a waste of time that I could replicate by hand.

The new Final Cut Studio’s Motion has audio analysis (plus a bunch of other things its ripping off from Shake, such as optical flow) so I may well end up trying it again in Motion instead of Shake.

All in all though, the technique worked — The squiggles squiggle in time to the bass, which looks cool, like the bright part of the fractal is shaking. Neat.

The final node mode includes more than just the squiggle. The original image had black parts in the “lake” that didn’t look good, so I ran things through a clamp to limit how black things got. Then I isolated the bright parts of the image with a LumaKey. One part of the node tree then warped it, as just described; the other part of the node tree blurred it to create a bright glow. Here’s the tree. Looks like fun, eh?

And here’s the before and after — before is on top, final image for video is on the bottom:

The main thing you notice in the video, though, isn’t the squiggles, it’s the fractal morphing in time to the theme. This was done by hand-setting keyframes in UF. Aargh, aargh, aargh. Painful, and improbable to want to change anything when you’re done. I created a spreadsheet to correlate the frame with the beat of the music. I then inserted each keyframe into UF. This took about six hours and lots of double checking. Talk about a waste of time! Study #5 will address this — I will have something that reads the midi file and inserts keyframes automatically. This will allow me to render stuff at low quality, compose music, re-render with new keyframes, re-score and adapt the music, re-render, etc until things are settled down and I do a final mixdown and high-res render. There’s no way I’d iterate if I’m doing this stuff by hand.

The last bit I did was bring the opacity of one of the layers up and down to relate to the middle section and the lead at the end. This was done by hand in Final Cut, simply by listening to the clip playing and hitting M (for “Mark”) at every musically significant point, then going back and hand-inserting keyframes around those marks. Total time was about 1 hour.

Music

Once again, an after-thought, which is weird because I’m really a composer looking for an excuse to compose and have fallen into making a fractal video to provide that excuse.

I was delighted with the focus that having visuals provided to my composing. I’m reminded that in college, some of my best music was done for the incredibly talented dancer and choreographer Sherry Kohn — having a visual in mind is a really powerful thing to rein in the music. Great music is composed by limiting yourself to a handful of things and exploring them rather than having a completely blank slate. Although this little piece wasn’t evenly remotely as consuming as what I’ve done in the past, and what I imagine I’ll be doing for the movie, it was still cool to be able to interact with the video. I think it’s sort of like working with another musician — you get more creative with another person. Even though I’m interacting with myself here, it’s in a totally different medium, so it ends up being synergistic.

The fractal I was playing with screamed out “technoish” to me. Whipped this out by diddling out the bassline first, then adding drums, then wondering what to do. Decided that emptying things out for a couple of measures and then bringing in a new voice would be a reasonable path forward. Rapidly ran out of ideas about when the fractal animation ended. Phew!

Total time on music: maybe 2 hours to compose, 1 hour to mix. Can hardly wait to spend more time on the music part of this. As I said, I’m much more of a composer than a visual artist.

Conclusion

This was a tiring and trying study. I’m glad it’s done. I think I’m likely to collapse into a gibbering heap when I finish the movie. This stuff is non-trivial.

I really need to clone myself. Maybe that’ll be Study #11. Requires research into leading edge biotech, but hey, I’m in Seattle, so I’ve got the companies around me…

I’m convinced that changing multiple parameters of the fractal, and when appropriate, in time to the music, is critical to maintaining interest. I know now that it can be done, but I also know now that writing a midi pre-processor is absolutely key to not going insane doing it. Wish me luck!