Possibility Spaces

Well, someone pointed out that I haven't updated this in a while (Hi, Biju!), and I did have a thought...

I was stuck in a middle seat, on a flight from Montreal to Vancouver yesterday, so I wound up playing a bunch of Solitaire on my Palm LifeDrive. I never really cared for the game - I thought it was like crossword puzzles, and entirely mechanical/deterministic. But it's not, and there are a bunch of different choices you make as the game goes on that can determine whether you win or lose (an "undo" function on my game let me back out of several losing games and follow a different path to win).

Anyway, it got me thinking about the different branches you can take in the game (possibility spaces), and ways to weight the winning branches, create a winning algorithm - standard fare for genetic algorithms.

That got me thinking about the possibility spaces for queuing algorithms - trying to maximize throughput of a system using a heuristic based on recurring types of meta data in the job submissions. You can't "undo" queuing decisions, but you can change them over time based on historical results, and optimize them. Not something super-relevant for a production queuing system, since there are usually far more production/political issues that need to be weighed, than strict efficiency concerns.

But that got me thinking about possibility spaces for business deals (and, really, personal interactions of all sorts) - where there is no undo, and the historical conditions from case to case are so different that it's hard to create a heuristic beyond "work with people you trust, and get it agreed to in writing - for everybody's sake".

In retrospect, I suppose that's only really interesting if you're currently weighing different business choices - but at least it's a new blog entry! More to follow.

How Would Google Render?

Here's another interesting post from Tim O'Reilly: "What Would Google Do"? Render farm management isn't a "Live" application (yet), but there are still plenty of lessons to be taken from Google, or Amazon. (Amazoogle!)

In my last post, I held Amazon up as the only real contender for an online compute/render service; Google would be the other obvious choice, though it doesn't offer a general computing service (yet).

But it's the history-tracking possibilities that interest me, more than the "Live" aspect of a web service. We already use a MySQL database behind Qube! - which lets you track the history of everything that goes through the farm. This kind of history is extremely useful for figuring out future bids and purchases: what was our peak/average utilization? How many iterations of each shot are we doing? What's our average render time? In short - where are we spending our time and money, and where are the bottlenecks?

And there's still more room for improvement. Optimal scheduling is a difficult problem, that can be made easier when you know more about the items in the queue: in particular, how long will they take to run? (Shortest job first is an easy/obvious optimization, but only if you know which jobs are the shortest.) But digital media is extremely iterative - even though the parameters of the shot will change between iterations, there's a lot of predictive information that can be passed from job to job. Texture references can be cached on the local disks of the render nodes - and their presence can be used as a weighting factor to favor jobs that can use them. "Hotspots" on central storage can be "cooled" by favoring jobs that don't draw from those areas - lots of possibilites.

The data is relatively easy to collect, and not overly difficult to incorporate into the decision-making process; the real trick is being able to build an interface that's simple enough to be generally useful. Queuing systems need to be easier to use, and easier to set up - and advanced functionality can't come at the expense of that useability...

A failure to communicate...

I look at a lot of grid/cluster/render farm services. I haven't seen a lot of big success stories on the media and entertainment front, for a number of reasons:

- Bandwidth: render jobs have relatively large input/output files, and the cost of moving the files back and forth can be relatively high, compared to the cost of computation.
- Security: most studios are very concerned about leaks of imagery or data; interestingly, I've heard of large studios that farm out work to smaller boutique studios - who then farm their renders out to render services...
- Cost: Visual FX comanies tend to be much more sensitive than other industries (why that's so will have to be the topic of another post); "grid" services that charge an $1/CPU-hour (bought in bulk, in advance) are prohibitively expensive.
- Lack of persistent storage: It's not efficient to upload separate copies of shared source files (referenced geometry and textures) for every render. Shared persistent storage is the optimal solution, but it's not available with all grid services - and when it is, there's an additional cost.
- Application licenses, and proprietary code: the cost of 3rd-party apps (e.g. Maya, Mental Ray, Renderman) frequently exceeds the cost of the hardware it runs on, the license EULAs for these apps usually prohibit any kind of rental or service usage, and there's no persistent connection (e.g. VPN) to share local licenses. Also, advanced render pipelines almost always include proprietary, site-specific code, which presents a similar problem.

Amazon's EC2 service may be the exception - the issue of application licenses remains, but Amazon's S3 storage service solves most of the storage issues. If Amazon could provide some kind of persistent VPN connection, we'd be set. Their costs (currently 10 cents per CPU-hour for computation, and 15 cents per GB/month of storage) are much more competitive; the transfer costs for S3 (10 cents per GB uploaded, 13-18 cents per GB downloaded) are more porblematic, but at least it's getting closer.

A bigger question is, what does this mean for the business models for storage, servers, and application software? In the short term, nothing. In the longer term, I think this is a business model that will have to be addressed...

Synchronicity III

I was all set to write about the NAB trip (and subsequent LA visit) and the synchronicity of trade shows and business travel - sometimes things just seem to come together.

But this morning I was sitting out on the patio, keeping an eye on my daughter, playing in her new sandbox - just enjoying the sun, and catching up on some reading. And just having an hour or so of "down time" gave me more clarity, more ideas, and more opportunity to make 'connections' - in my head, and with what I'd been reading - than a week of business travel, trade shows, and meetings.

Design Failure

So, I'm headed back to Vegas for NAB, and I got asked to change my ticket from Sunday to Saturday, get there a day early for some meetings. Unfortunately, I bought my ticket through Cheap Tickets, and while I can change that flight, I would also have to change my flight the next day, because it's sold out...

"But I'm already booked on that flight..."
- "Yes sir, but I have to issue an entirely new itinerary, and I can't do that because your second flight is sold out."
"But I don't want to change my second flight - I'm travelling with someone on that flight..."
- "Then you can't change your first flight."

I talked to the airline, and they couldn't do anything - had to be changed by Cheap Tickets. So I said okay, I'll book another flight, on a different airline, and just eat the other flight. Did that, and called Cheap Tickets for cancel the first leg:

- "I'm sorry sir, this is a round-trip flight, and you can't cancel the first leg."
"Okay, well, I just won't be taking it - I'm taking another flight down the day before - but then I'll complete the rest of my ticket, as planned.
- "No sir, if you aren't on your first flight, we automatically cancel the rest of your flights."

How is it possible to do business like this? How can somebody have an interface so crippled, so useless, that it's not possible to make a simple change like this? It can't be that they don't want to be able to do it - they were going to charge $150 US in change fees.

Never assume malice when stupidity will suffice.

Crossing a line

I've enjoyed Kathy Sierra's blog, Creating Passionate Users, quite a number of times, and I was very disturbed to read about her recent troubles. How is it that the RIAA can get ISPs to roll over on alleged 10-year-old file-downloaders, but nobody will roll over on someone making death threats? I don't want to get into a long and pointless discussion about the way people behave online, but I will say this: Kathy, don't let the bastards get you down.

Iteration, iteration, iteration!

This post at Gamasutra caught my eye. I didn't make it to GDC this year (I was at OTC, as I noted in my last post), so I missed the actual talk, unfortunately. But "iteration wins" because a big, collaborative effort (like a game, or a film) has everything to do with a tenacious attention to detail, and the willingness (and patience) to tweak and refine the details until everything is "just right". A singular vision is essential - a director or a designer who can give clear, concise direction to the artists - but pipeline is the way that happens. And the faster your pipeline works, the faster you can iterate, and the more iterations you get, the better your project turns out - it really is that simple.

Vegas, Baby!

I spent last week in Las Vegas at Autodesk's One Team Conference (It's over now, but you can read about it here.) It was good - and thing that I thought was interesting was, it was a (mostly) internal conference. It's not about meeting up with customers, it's about Autodesk connecting with their resellers and partners, helping them be more successful, and hearing about the issues those resellers encounter. Yes, it's important to meet with customers, and listen to them - and that's what we all need to do, every day - but I think every company could benefit from looking inward, and trying to improve their processes, at least once in a while.

I was in Toronto this week, and I got a chance to have dinner with a bunch of guys - some old friends, some new - with several decades of experience in animation film and TV, between us. We spent a couple rounds of drinks regaling one another with horror stories about productions gone wrong: why does it seem like everybody makes the same mistakes, trying to set up a pipeline? Part of it seems to be the relative immaturity of the industry - it just hasn't been around long enough - nor does it stay still long enough - for "best practices" and standards to emerge. In some ways, this is a good thing - there's a lot of competitive advantage to be had in pipeline, and companies that "coast" are going to find their work outsourced to someone better/faster/cheaper. But for all of it's importance, building pipelines doesn't seem to get the same attention as other aspects of the industry - aside from one or two "Birds of a Feather" meetings at SIGGRAPH, there are virtually no discussion groups, or papers or panels or even online discussion groups, dedicated to the art and science of building production pipelines. Larger studios usually have a pipeline architect (qualified or not!), but smaller studios often relegate it to "the tech guy" - maybe an artist who does some scripting, too, or the IT guy who maintains the email and DNS servers. Worse yet is the studio who gets a consultant - one that hasn't actually worked on a production, no less - to come in and really screw things up; I'm in the middle of fixing one of those, now...

Logotherapy

The gods had condemned Sisyphus to ceaselessly rolling a rock to the top of a mountain, whence the stone would fall back of its own weight. They had thought with some reason that there is no more dreadful punishment than futile and hopeless labor. - Albert Camus

Most of the work in making a film, or a game, is iterative - constantly refining every aspect of it, tweaking and adjusting and refining all the details. If that process of iteration - the steps by which each aspect of the project is refined - is not itself refined, then this can be a tedious and labored process. The usual motivations for refining a pipeline are monetary - if you can shorten the iterative loop, you can get more loops (a better product) in the same amount of time, or a set number of loops in a shorter time: either of which increases the 'efficiency of improvement'.

Pipeline engineering is a kind of meta-aesthetic activity: the process of refining a process. But it can also have a huge aesthetic aspect - just like any tool can be made rough and clumsy, or refined and elegant. I've seen a lot of pipelines, a lot of workflows, that were inefficient or inelegant, usually in the name of expediency -that's not only a flawed way of thinking, it's downright inelegant.

First things First

How do people manage to work together on large, creative, aesthetically-driven projects? Games and animated films and visual effects projects are - more often than not - the result of the combined efforts of a lot of creative, technical, and artistic people. They also produce a lot more wasted effort, wasted time, pain and suffering than they need to - more often than not.

Mostly, that's what I plan to write about - how these things get done, the mistakes that get made, the improvements that happen along the way, what's happening in the digital media industries, what's happening to the people that work in it, and where I think it's all headed.

Other topics will include things like: running a business, the perils of consulting, dealing with difficult people, venture capital, the difference between activity and results, why it's hard to get a good pizza in Hawaii. You are invited to join the conversation.