Joe Letteri of Weta Talks Way of Water

While the ultimate success of Avatar: The Way of Water is still being determined, it’s already changed filmmaking as we know it. Long before the sequel hit theaters, director James Cameron invested so much time and Hollywood money into the franchise, it gave the team at Weta Digital the confidence and resources to upgrade the way they made not just Avatar, but all movies.

So, thanks to Avatar: The Way of Water, Joe Letteri and his team at Weta were able to use a brand new system for facial animation, making the Na’vi of Pandora even more believable and expressive than they were in the first film. They pioneered a unique rig to make sure CGI characters and humans could get the right eye line. They built a whole new system to keep data flowing from the set and into Weta computers. The list goes on and on.

io9 spoke with Letteri, the film’s senior VFX supervisor who won Oscars for his work on Avatar, King Kong, and The Lord of the Rings: The Return of the King as well as The Two Towers, about the decade-plus build-up to Avatar: The Way of Water.

This interview has been edited for length and clarity.

Germain Lussier, io9: So you began working on this film in 2017 but the press notes say, in 2010, James Cameron had a tech summit to discuss the sequels then. What came out of that meeting?

Joe Letteri: So what came out of that was basically, Jim convened it to say, “Okay, when we do the next one”— which was good because it meant we were doing a next one—“When we do the next one, how do we do this better?” One of the things that happened on the first one, [Cameron’s company] Lightstorm was spinning up this whole process: their virtual production, their stage, their art department…and because the film had not actually been greenlit—this is like back in like 2006—[Weta] couldn’t officially come on and work with them to build an integrated system. So we gave as much advice as possible but really we discovered, going through the process, a lot of the data that came through had to be redone because you’re doing something in one system and moving it to another system. And we said the first thing we have to do [on a sequel] is integrate what we’re doing on both sides and Lightstorm agreed. So we immediately sat down and started to build a system that would work on both ends.

All the things we saw in the first film that we thought we could do better, we just started working on. And, fortunately, we were doing a lot of other films in that period that we could iterate on the software and try it and use it. These things became necessary to do the other films we’ve been working on as well.

io9: So you’re working on these other movies thinking, “Oh, yes, this is going to work on Avatar 2.”

Letteri: Exactly right. And I think the last step on that train for us was Alita: Battle Angel. Because we were working with Lightstorm directly, even though it was Robert Rodriguez directing it. It was a lot of the same teams, a lot of the same tools. Like, for the onset capture, we really battle-tested for the last time that suite of tools before we cut loose with Jim on it.

io9: So it’s been about five years since your team started working only on Avatar 2, which is a lifetime in terms of technological advancement. Is there ever a situation where you’re using a piece of technology or a piece of software and you realize as the shot is almost complete that there’s new technology that can do it better?

Letteri: Anything that’s changing, we’re writing a lot of it ourselves. The new facial system, the new simulation engine, that’s us doing it. So we kind of know where we are. And at that point, we know if we’re close to having something better, we’ll hold off finishing the shot because if you can make it better, you want to make it better. So we didn’t really get caught out by like “Hey, let’s go back and look at what we did two years ago and let’s see if we can do it better.”

io9: So of all the things your team created for this film, what was the biggest advancement? Was it that integrated pipeline between Weta and Lightstorm?

Letteri: Our facial animation system coming off the heels of Alita. Alita was still using a version of the system that we had created for Gollum. It had been refined and gone through a lot of iterations and it was based on a FACS system— “facial action coding system”—and it served us well. But we realized the kind of detail we were chasing in Alita, and that was just one single character, that system had kind of reached the end of its life. And so I started thinking about a new way that we might do it, and hit on the idea that we could actually use a neural network to mimic the way that the muscles in the face behave. So that’s what we did. We wrote a system that was based actually on a series of neural networks to kind of transcode, if you will, or transcribe what we’re seeing an actor doing into what the character is doing—and allowing for hooks of either running a physical solver on it or having animators directly manipulate it. Everybody was working with the same under-the-hood control structure, which is understanding how the muscles drive emotion.

io9: If you had come up with that idea on the first Avatar, or any other film you’ve done since, was the technology there to do it before or is that something that is specifically a 2022 thing?

Letteri: I think pretty much it was [a new thing] because it’s a difficult system to spin up. So you need the right project to do it. You need a project where you know you’re going to have a large number of characters in close up and you might be doing more than one movie. That’s where it’s worth taking the leap to do it. Otherwise, even though the older system is harder, you’ve got animators who still know how to make it work and that may actually be more efficient. You can still get good results out of it. But it would have been pretty much impossible to get the results we got on this film using that old system.

io9: Right. I think people don’t always understand when they talk about however much money a movie like this costs, it’s not only going to the actors or digital effects. Sometimes it’s people creating software or technology that literally didn’t exist before, and that can be expensive.

Letteri: Yes, exactly.

io9: Obviously this is a visual effects-driven film but there are lots of on-set innovations happening too, such as underwater performance capture. How crucial is it to get the best possible on-set raw materials before your team takes a crack at it?

Letteri: One hundred percent, and you’ve got two aspects there. One is performance. If you’ve got an actor’s performance, yes, you can go in and change it, even if it’s a real actor after the fact. But you really don’t want to. You want those beats that are in the moment. If you don’t like it, just get another take. That’s the time to do it, you know what I mean? But what in the past has been really tricky is live-action characters dealing with CG characters. Eye lines are the biggest issue. And we knew we wanted to get past like the tennis ball on a stick. So we created an eye line system that was basically a cable cam so we could take the performance of the virtual actors and run it through this cable cam. And rather than just having a tennis ball there, we had a monitor there that showed the actor’s performance.

So if you take Spider, for example. Because Jim does this personal performance capture step, he shot the whole movie in performance capture. So Jack Champion [who plays Spider] was working with Stephen Lang [Quaritch] and they played these scenes already. So when it came time for Jack to do it live, he’s on set, but he’s got this monitor now that’s moving around at Quaritch’s correct eye position, but with Stephen’s performance on it. So he’s reacting to the same performance that he’s already used to working with. And so it gave us eye line. It gave us movement through space that was correct. And you’re not doing the usual thing where the actor is chasing the tennis ball with the guy that’s holding it. Or the character can’t actually walk that fast or the character has to bend over. Like [in the past], you’re always having to make these animation adjustments to try to make it believable. But now, because everything was in the right place to begin with, your camera framing is there because we had this new depth compositing tool that we wrote. So you’re seeing everything integrated live pixel by pixel in proper 3D as you’re shooting it. It gave us a much stronger baseline to integrate the live action with.

io9: Was there a particular shot or scene or character that was the most difficult or that you knew, if you got this right, everything else would be fine?

Letteri: The very first scene we did was the dialogue scene with Jake and Neytiri where they’re in High Camp and Jake is trying to convince her to leave. And that was really bringing back the two characters from 15 years earlier, dusting them off, updating them to all the new software, the new detail in the skin, the new lighting techniques, the new hair, but also the new facial system. And that was really where we battle-tested pretty much everything except for the water, in that one scene. So we actually spent a year on that scene just to make sure we had it right and understood it. At that point, we started rolling it out to the other characters and, you know, branching out to do the rest of the shots.

io9: I’m surprised you didn’t say something from the final battle sequence because it’s so incredible. How does that scene compare to other things you’ve done?

Letteri: I think this is the biggest project I’ve worked on because of the water. Like when a character hits the water, they move differently. When they emerge from the water, they move differently. They’re interacting with the water and water is splashing and exploding everywhere. There’s a lot more complex when you’re doing so much interaction with water.

io9: Is there anything that you couldn’t do effects-wise before this movie but can do now after it?

Letteri: Jim always says, “Hey, by the time we’re done, we’ll know how to make this movie.” And that was about right, especially when it came to the water. Like we had the characters on a pretty good roll early on. But the water just kept getting bigger and bigger, especially as we were doing the battle scenes. We saved some of those bigger scenes until we really had a better idea of how to deal with the water under our belts. So I think we’re in pretty, pretty good shape now to understand what we need to do better next time.

io9: Right. So because you spent so much time getting everything right on this one, how much easier are the subsequent sequels? 

Letteri: Well, I think certain things we won’t have to spend as much time on, like the characters. I know there are things that I want to change that we can do better there, but we’ve got a pretty good basis for it. Same thing with the water. There are things we can do better there. We’ve got a pretty good basis for it, but we’ll fill up that time with all the new stuff in the story that Jim has come up with. So there’ll still be a lot to do. 

io9: In terms of an actor, having four or five scripts that reveal where your character is going or where it’s been can be very helpful. Is it at all helpful in visual effects to know where the story and the characters are going so far in advance?

Letteri: Yeah, it really does for kind of a combination of the two reasons we just talked about. One is, you know, where you really can put your time and effort and invest in getting these systems going because you know they’re going to carry you across. You might make a different decision if the timeframe you’re going to use it in is very short, but also in the fact that then you can build on it. Like, you know, a 13-year gap is different than a three-week gap. So now we’re rolling right into the next film and you have that momentum behind you. The artists know what to do and actually, you don’t want to stop because you want to you don’t want to lose that momentum.

io9: How important is it that the team on these films largely remains the same, if they in fact do?

Letteri: Absolutely. Look, I would say at least—maybe not half—but there were several hundred people on this film that were around for the first film, and there were at least 100 who were around on [Lord of the] Rings. Like we’ve got good longevity with our team here. We’ve been together for a long time. There’s good depth in it. It really helps when you’re working because we know what we’re talking about. You can really turn these things around a lot quicker.

io9: That leads into something else I’m curious about. I feel like with Weta being so close-knit and so far away, it might avoid some issues visual effects artists have been talking about in recent years. There are lots of stories about the effects industry, people being overworked, underpaid, and should there be unions? As someone who has been doing this for so long, and won multiple Oscars, I’m curious about your take.

Letteri: Yeah. The overwork thing, I don’t know how you get around that. I mean, one of the things that would be helpful is having a little better plan on some films coming into it. You hear a lot of stories about, “Okay, we’re making up the third act as we go.” Fortunately, that wasn’t the case in this one. But you still work hard because you only get one bite at the apple, right? Like, if you go home early and you don’t put in that one thing that you really want to put in, it’s not going in the movie forever, you know? This is what we do. This is your opportunity to get it in there. You sleep next week. Look, we’ve all pulled all-nighters for worse causes than this so that’s why a lot of us get into this business.

But I also understand that a lot of people don’t want to be forced to do that all the time because the industry has been cranking for the last, you know, 10 years or so. So people do need a break and they need to be covered when they get that break. You got to take care of that as well. I mean, you want your artists always to be performing at their peak and their peak is going to be whatever makes them happy because it’s a very creative, collaborative business. You got to have that.

io9: So stepping back a bit, Avatar came out in 2009 and changed the game. This movie, 13 years later, is changing the game again. Avatar 3, though, will be out in two years. So how, or can, these movies be as technologically important as the first two, or do they not have to?

Letteri: I don’t think they have to. I think that’s a byproduct of us creating what we need to do to see what we want to see on screen. It’s not about driving the technology so much as, like, we have to build it because we need it.

io9: Jim mentioned recently that an Avatar TV show could happen in the future if the technology got cheaper. And I know that’s something George Lucas said about Star Wars—and now, finally, we have Star Wars TV. So what specifically do you think would have to change for an Avatar TV show to happen to be cost-effective?

Letteri: Well, the thing about doing feature films that’s different than TV is feature films, you take the time to develop these new things. So when you say something is expensive, like what you were saying earlier, there’s a lot of costs that goes into spinning these things up for the very first time, right? I remember working on Jurassic Park. We did like 65 shots and it took a year. That was a big deal. Now you can do a dinosaur on your home computer. But if you’re doing something for the first time, it’s going to take time. And a lot of TV shows don’t have that luxury because of the time frame they’re on. Film is all about taking that time. Like I said, our first scene took a year. How many episodes would you have to deliver in a year? It’s a different time scale, and that’s why I like working in film, because we get to take that and, you know, build it.

Avatar: The Way of Water is now in theaters.

Want more io9 news? Check out when to expect the latest Marvel, Star Wars, and Star Trek releases, what’s next for the DC Universe on film and TV, and everything you need to know about James Cameron’s Avatar: The Way of Water.

While the ultimate success of Avatar: The Way of Water is still being determined, it’s already changed filmmaking as we know it. Long before the sequel hit theaters, director James Cameron invested so much time and Hollywood money into the franchise, it gave the team at Weta Digital the confidence and resources to upgrade the way they made not just Avatar, but all movies.

So, thanks to Avatar: The Way of Water, Joe Letteri and his team at Weta were able to use a brand new system for facial animation, making the Na’vi of Pandora even more believable and expressive than they were in the first film. They pioneered a unique rig to make sure CGI characters and humans could get the right eye line. They built a whole new system to keep data flowing from the set and into Weta computers. The list goes on and on.

io9 spoke with Letteri, the film’s senior VFX supervisor who won Oscars for his work on Avatar, King Kong, and The Lord of the Rings: The Return of the King as well as The Two Towers, about the decade-plus build-up to Avatar: The Way of Water.

This interview has been edited for length and clarity.

Germain Lussier, io9: So you began working on this film in 2017 but the press notes say, in 2010, James Cameron had a tech summit to discuss the sequels then. What came out of that meeting?

Joe Letteri: So what came out of that was basically, Jim convened it to say, “Okay, when we do the next one”— which was good because it meant we were doing a next one—“When we do the next one, how do we do this better?” One of the things that happened on the first one, [Cameron’s company] Lightstorm was spinning up this whole process: their virtual production, their stage, their art department…and because the film had not actually been greenlit—this is like back in like 2006—[Weta] couldn’t officially come on and work with them to build an integrated system. So we gave as much advice as possible but really we discovered, going through the process, a lot of the data that came through had to be redone because you’re doing something in one system and moving it to another system. And we said the first thing we have to do [on a sequel] is integrate what we’re doing on both sides and Lightstorm agreed. So we immediately sat down and started to build a system that would work on both ends.

All the things we saw in the first film that we thought we could do better, we just started working on. And, fortunately, we were doing a lot of other films in that period that we could iterate on the software and try it and use it. These things became necessary to do the other films we’ve been working on as well.

io9: So you’re working on these other movies thinking, “Oh, yes, this is going to work on Avatar 2.”

Letteri: Exactly right. And I think the last step on that train for us was Alita: Battle Angel. Because we were working with Lightstorm directly, even though it was Robert Rodriguez directing it. It was a lot of the same teams, a lot of the same tools. Like, for the onset capture, we really battle-tested for the last time that suite of tools before we cut loose with Jim on it.

io9: So it’s been about five years since your team started working only on Avatar 2, which is a lifetime in terms of technological advancement. Is there ever a situation where you’re using a piece of technology or a piece of software and you realize as the shot is almost complete that there’s new technology that can do it better?

Letteri: Anything that’s changing, we’re writing a lot of it ourselves. The new facial system, the new simulation engine, that’s us doing it. So we kind of know where we are. And at that point, we know if we’re close to having something better, we’ll hold off finishing the shot because if you can make it better, you want to make it better. So we didn’t really get caught out by like “Hey, let’s go back and look at what we did two years ago and let’s see if we can do it better.”

io9: So of all the things your team created for this film, what was the biggest advancement? Was it that integrated pipeline between Weta and Lightstorm?

Letteri: Our facial animation system coming off the heels of Alita. Alita was still using a version of the system that we had created for Gollum. It had been refined and gone through a lot of iterations and it was based on a FACS system— “facial action coding system”—and it served us well. But we realized the kind of detail we were chasing in Alita, and that was just one single character, that system had kind of reached the end of its life. And so I started thinking about a new way that we might do it, and hit on the idea that we could actually use a neural network to mimic the way that the muscles in the face behave. So that’s what we did. We wrote a system that was based actually on a series of neural networks to kind of transcode, if you will, or transcribe what we’re seeing an actor doing into what the character is doing—and allowing for hooks of either running a physical solver on it or having animators directly manipulate it. Everybody was working with the same under-the-hood control structure, which is understanding how the muscles drive emotion.

io9: If you had come up with that idea on the first Avatar, or any other film you’ve done since, was the technology there to do it before or is that something that is specifically a 2022 thing?

Letteri: I think pretty much it was [a new thing] because it’s a difficult system to spin up. So you need the right project to do it. You need a project where you know you’re going to have a large number of characters in close up and you might be doing more than one movie. That’s where it’s worth taking the leap to do it. Otherwise, even though the older system is harder, you’ve got animators who still know how to make it work and that may actually be more efficient. You can still get good results out of it. But it would have been pretty much impossible to get the results we got on this film using that old system.

io9: Right. I think people don’t always understand when they talk about however much money a movie like this costs, it’s not only going to the actors or digital effects. Sometimes it’s people creating software or technology that literally didn’t exist before, and that can be expensive.

Letteri: Yes, exactly.

io9: Obviously this is a visual effects-driven film but there are lots of on-set innovations happening too, such as underwater performance capture. How crucial is it to get the best possible on-set raw materials before your team takes a crack at it?

Letteri: One hundred percent, and you’ve got two aspects there. One is performance. If you’ve got an actor’s performance, yes, you can go in and change it, even if it’s a real actor after the fact. But you really don’t want to. You want those beats that are in the moment. If you don’t like it, just get another take. That’s the time to do it, you know what I mean? But what in the past has been really tricky is live-action characters dealing with CG characters. Eye lines are the biggest issue. And we knew we wanted to get past like the tennis ball on a stick. So we created an eye line system that was basically a cable cam so we could take the performance of the virtual actors and run it through this cable cam. And rather than just having a tennis ball there, we had a monitor there that showed the actor’s performance.

So if you take Spider, for example. Because Jim does this personal performance capture step, he shot the whole movie in performance capture. So Jack Champion [who plays Spider] was working with Stephen Lang [Quaritch] and they played these scenes already. So when it came time for Jack to do it live, he’s on set, but he’s got this monitor now that’s moving around at Quaritch’s correct eye position, but with Stephen’s performance on it. So he’s reacting to the same performance that he’s already used to working with. And so it gave us eye line. It gave us movement through space that was correct. And you’re not doing the usual thing where the actor is chasing the tennis ball with the guy that’s holding it. Or the character can’t actually walk that fast or the character has to bend over. Like [in the past], you’re always having to make these animation adjustments to try to make it believable. But now, because everything was in the right place to begin with, your camera framing is there because we had this new depth compositing tool that we wrote. So you’re seeing everything integrated live pixel by pixel in proper 3D as you’re shooting it. It gave us a much stronger baseline to integrate the live action with.

io9: Was there a particular shot or scene or character that was the most difficult or that you knew, if you got this right, everything else would be fine?

Letteri: The very first scene we did was the dialogue scene with Jake and Neytiri where they’re in High Camp and Jake is trying to convince her to leave. And that was really bringing back the two characters from 15 years earlier, dusting them off, updating them to all the new software, the new detail in the skin, the new lighting techniques, the new hair, but also the new facial system. And that was really where we battle-tested pretty much everything except for the water, in that one scene. So we actually spent a year on that scene just to make sure we had it right and understood it. At that point, we started rolling it out to the other characters and, you know, branching out to do the rest of the shots.

io9: I’m surprised you didn’t say something from the final battle sequence because it’s so incredible. How does that scene compare to other things you’ve done?

Letteri: I think this is the biggest project I’ve worked on because of the water. Like when a character hits the water, they move differently. When they emerge from the water, they move differently. They’re interacting with the water and water is splashing and exploding everywhere. There’s a lot more complex when you’re doing so much interaction with water.

io9: Is there anything that you couldn’t do effects-wise before this movie but can do now after it?

Letteri: Jim always says, “Hey, by the time we’re done, we’ll know how to make this movie.” And that was about right, especially when it came to the water. Like we had the characters on a pretty good roll early on. But the water just kept getting bigger and bigger, especially as we were doing the battle scenes. We saved some of those bigger scenes until we really had a better idea of how to deal with the water under our belts. So I think we’re in pretty, pretty good shape now to understand what we need to do better next time.

io9: Right. So because you spent so much time getting everything right on this one, how much easier are the subsequent sequels? 

Letteri: Well, I think certain things we won’t have to spend as much time on, like the characters. I know there are things that I want to change that we can do better there, but we’ve got a pretty good basis for it. Same thing with the water. There are things we can do better there. We’ve got a pretty good basis for it, but we’ll fill up that time with all the new stuff in the story that Jim has come up with. So there’ll still be a lot to do. 

io9: In terms of an actor, having four or five scripts that reveal where your character is going or where it’s been can be very helpful. Is it at all helpful in visual effects to know where the story and the characters are going so far in advance?

Letteri: Yeah, it really does for kind of a combination of the two reasons we just talked about. One is, you know, where you really can put your time and effort and invest in getting these systems going because you know they’re going to carry you across. You might make a different decision if the timeframe you’re going to use it in is very short, but also in the fact that then you can build on it. Like, you know, a 13-year gap is different than a three-week gap. So now we’re rolling right into the next film and you have that momentum behind you. The artists know what to do and actually, you don’t want to stop because you want to you don’t want to lose that momentum.

io9: How important is it that the team on these films largely remains the same, if they in fact do?

Letteri: Absolutely. Look, I would say at least—maybe not half—but there were several hundred people on this film that were around for the first film, and there were at least 100 who were around on [Lord of the] Rings. Like we’ve got good longevity with our team here. We’ve been together for a long time. There’s good depth in it. It really helps when you’re working because we know what we’re talking about. You can really turn these things around a lot quicker.

io9: That leads into something else I’m curious about. I feel like with Weta being so close-knit and so far away, it might avoid some issues visual effects artists have been talking about in recent years. There are lots of stories about the effects industry, people being overworked, underpaid, and should there be unions? As someone who has been doing this for so long, and won multiple Oscars, I’m curious about your take.

Letteri: Yeah. The overwork thing, I don’t know how you get around that. I mean, one of the things that would be helpful is having a little better plan on some films coming into it. You hear a lot of stories about, “Okay, we’re making up the third act as we go.” Fortunately, that wasn’t the case in this one. But you still work hard because you only get one bite at the apple, right? Like, if you go home early and you don’t put in that one thing that you really want to put in, it’s not going in the movie forever, you know? This is what we do. This is your opportunity to get it in there. You sleep next week. Look, we’ve all pulled all-nighters for worse causes than this so that’s why a lot of us get into this business.

But I also understand that a lot of people don’t want to be forced to do that all the time because the industry has been cranking for the last, you know, 10 years or so. So people do need a break and they need to be covered when they get that break. You got to take care of that as well. I mean, you want your artists always to be performing at their peak and their peak is going to be whatever makes them happy because it’s a very creative, collaborative business. You got to have that.

io9: So stepping back a bit, Avatar came out in 2009 and changed the game. This movie, 13 years later, is changing the game again. Avatar 3, though, will be out in two years. So how, or can, these movies be as technologically important as the first two, or do they not have to?

Letteri: I don’t think they have to. I think that’s a byproduct of us creating what we need to do to see what we want to see on screen. It’s not about driving the technology so much as, like, we have to build it because we need it.

io9: Jim mentioned recently that an Avatar TV show could happen in the future if the technology got cheaper. And I know that’s something George Lucas said about Star Wars—and now, finally, we have Star Wars TV. So what specifically do you think would have to change for an Avatar TV show to happen to be cost-effective?

Letteri: Well, the thing about doing feature films that’s different than TV is feature films, you take the time to develop these new things. So when you say something is expensive, like what you were saying earlier, there’s a lot of costs that goes into spinning these things up for the very first time, right? I remember working on Jurassic Park. We did like 65 shots and it took a year. That was a big deal. Now you can do a dinosaur on your home computer. But if you’re doing something for the first time, it’s going to take time. And a lot of TV shows don’t have that luxury because of the time frame they’re on. Film is all about taking that time. Like I said, our first scene took a year. How many episodes would you have to deliver in a year? It’s a different time scale, and that’s why I like working in film, because we get to take that and, you know, build it.

Avatar: The Way of Water is now in theaters.

Want more io9 news? Check out when to expect the latest Marvel, Star Wars, and Star Trek releases, what’s next for the DC Universe on film and TV, and everything you need to know about James Cameron’s Avatar: The Way of Water.