By Jennifer Walden
As HBO's The Last of Us series geared up for production in Calgary, Alberta in 2021, the film industry was still navigating the challenges of creating content during the Covid pandemic. 4x-Emmy-winning producer Greg Spence was tasked with building a bulletproof production plan for TLOU that could overcome the uncertainties of the time. "We didn't know if or when illnesses would pick up or people would become unavailable or whether or not quarantine restrictions would be a factor. We really needed to find as flexible a cloud-based solution as possible for editorial," he says. "I did some research and found that Remote Picture Labs has the most truly cloud-based remote editing system available," says Spence, who envisioned the show's editors working from home (if they preferred) or at a small cutting room in L.A, which was set up for the show and could be expanded to grow with the production.
Remote Picture Labs (RPL) offers modular cloud-based post-production workflow solutions custom-tailored to fit a production's specific needs, and their solutions are scalable to meet a production's changing needs from pre-production to final delivery – to address new additions to the creative team, handle increased processing demands, and adapt to other changes to the collaborative work environment. RPL offers remote access to editing and VFX workstations via ultra-secure Zero Client endpoint devices. Their secure platform provides clients with a fault-tolerant private cloud, ultra-fast 10Gb connectivity, and secure transport using Teradici PCoIP Ultra to ensure reliability and enable exceptional user experiences. Plus, they offer equipment rentals, from speakers, stands, and monitors to office chairs and desks.
Setting up shop in Calgary meant connecting the on-location teams to the post-production office in L.A. and making sure that everyone in the production pipeline was getting what they needed as quickly as possible. "Other remote workflow options require editors and assistants to remote control their desktops set up in studio locations. But RPL uses a cloud-based system where our Avid NEXIS is truly in the cloud. Our challenge on TLOU was whether we could get a fat enough internet pipe across all the little branches of the workflow to make that work for us," says Spence.
First, Spence needed to find a local post-production facility that could handle the dailies in Calgary. "As long as we could get our dailies into RPL's cloud-based editing system, I knew we would be in excellent shape, and Calgary has a great technical infrastructure and good internet," he says.
The production then took over a massive car storage facility in Calgary and converted the raw space into five cutting rooms as well as post-production offices. Spence hired a local post supervisor, Tom Dutton, to help set up the cutting rooms and do the initial tests with RPL during prep to make sure everything was working. Spence says, "The hardware for the internet in that converted space needed upgrading. I try to have as deep a knowledge as I can across all aspects of post-production, but internet server capacity, flow through, and lag time are not my areas of expertise. Fortunately, it's part of RPL’s. They asked our internet expert in Calgary what kind of equipment were they using, and what type of bandwidth, and then recommended the equipment we'd need. It was just a few little server boxes that needed to be upgraded. It was really affordable and RPL easily solved the issue. I'm grateful that internet capacity is not something that I need to learn a ton about as long as I have RPL on my side."
Even though RPL rents out equipment and office furniture, Spence turned to local suppliers in Calgary for that gear. "Once we realized what the shipping costs were going to be, to ship everything there and back, we just bought what we needed up there. We only needed to ship RPL's Zero Client endpoint devices. Since Season 2 of The Last of Us will be in Vancouver, we'll just move all that equipment to Vancouver and set it up again," says Spence.
VFX Editor Luke Botteron, who started on the show in July 2021, spent his first three weeks working remotely from home in Vancouver before joining the team in Calgary. His workstation setup consisted of speakers, monitors, keyboard, mouse, a separate TV for reviews, and a Zero Client endpoint device to connect to the editing software inside RPL's environment. "The RPL environment was already in place, so my first day was just an orientation of signing in, getting comfortable with it, and then building my VFX databases in there. The setup was simple. I could still have RPL running and just switch tabs, so I could go into Avid, find what I needed to do my mockup, send it to our VFX machine, whip up a mock up, send it back, and then drop it all on the same workstation without having a bunch of different rigs around. My setup in Calgary was the same. Once we moved to L.A., RPL was there with us and was able to build out our cutting rooms with their hardware," says Botteron.
Traveling to Canada at that time meant spending two weeks in mandatory quarantine, so the editors were required to stay inside their local apartments for the duration of it. Spence rented three, 2-bedroom condos in Calgary with high-speed internet access and converted the guest bedroom in each into a mobile cutting room. "RPL's system really helped us out. The editors were able to work in their apartments in Calgary during their quarantine, and should anyone get sick (we were testing often) but feel fine, they could continue to work. With the RPL workflow, the editors had the option to work in our Calgary cutting room, work in their Calgary apartment, work in our L.A. cutting room, or work from home. And all of our picture editors ended up using all of those options," says Spence.
Editor Timothy Good, ACE, who also went to Calgary to work on the show during the shoot, notes the secret to RPL's success is that it's easy to work from any location with a high-speed internet connection. Good says, "RPL uses a 10ZiG Zero Client endpoint device, and I could literally login to the RPL environment, into the Avid, from anywhere. You just connect your monitors, keyboard, and mouse to the 10ZiG box, and you're into your system. It was literally the easiest scenario possible. It was so simple. I'll keep coming back to them because I think simplicity is always best when simplicity works in a technical environment.” Good even switched apartments while in Calgary, moving up to the 16th floor of the building. He says, "This way of working allows you to be very limber because you can move around as you need to and not have to move a lot of equipment. I had an incredible view from my new apartment and could take in the amazing beauty of the mountains. I was in heaven!"
Editor Emily Mendez started on TLOU in September 2021, at the same time as Good, but she worked from home in L.A. and did not travel to Calgary. "I had the same setup as Tim – a Zero Client endpoint device connected to my monitors and keyboard. Eventually, when we moved into the post-production office in L.A., we were able to connect to the RPL environment using a PCoIP software client. The application asks you to enter your username, password, and the site that you're logging into. That's it. Then you were in an emulator with your remote workstation. It was super easy," she says.
After production wrapped in Calgary, post-production ramped up in the L.A. offices, greatly expanding the footprint there. Spence says, "We got into serious visual effects and had a lot of activity on the VFX side, so we needed a lot of bandwidth. We had some hiccups in our visual effects team's file-based, cloud-based workflows, but RPL helped our VFX team work out their software glitches within ShotGrid and with some of the file exchange software they were using with their vendors. We actually ended up having just as many challenges moving files around between our VFX coordinators inside the cutting room as we did moving files back and forth to Calgary. Whether you're moving files six feet or 600 miles, if you need a 10-gigabit capacity, you need it. RPL's modular structure is what makes it all possible and easy. Otherwise, it could be really overwhelming."
Since RPL's flexible approach to workflows prioritizes the editor's needs, they are completely open-minded and able to incorporate workstations outside of the RPL environment. As Botteron explains, "We worked exclusively in the RPL environment for the first few months. I was logging into RPL to a virtual machine that was designed for Avid. When doing mockups inside of Avid, there was no problem at all for playback. But some of the other VFX software is just better suited to be on a side computer, so I started using my own personal computer that was built for VFX to do a lot of the heavier lifting."
Botteron worked closely with Erik Beauchamp, Chief Technology Officer at RPL, to figure out the best solution for his VFX workflow needs. "Erik was there the whole way through, offering different suggestions for what to try. We found that when I needed more processing power, it was easier to go offline. For instance, if I was building a world in Unreal Engine, it was much easier to do that on a designated computer. But if I was just helping Tim and Emily with a split comp, it was super easy to use the RPL machine for cleaning up the roto and doing those types of tasks. We had a ton of flexibility, but we definitely found the limits of what the RPL machines do based on how we built them for this show. But they're continually evolving their approach. Going forward, RPL has bigger plans for how they would approach VFX workflows rather than how we had set it up originally for The Last of Us," says Botteron.
Good concurs, "Erik was a fantastic liaison. He was always upgrading the software for us and making upgrades that would improve the sync. Another thing I really love about the RPL system versus every other remote system I've used is the sync and responsiveness. Sync is so critical to editors. Even a frame out of sync during playback makes it really hard to work. Erik would constantly make improvements to achieve even tighter sync, but it was never an issue. Not only that, the system was extremely stable. I don't remember crashing, but I'm sure it was only once or twice over a year and a half."
When an issue did arise, Botteron notes that Beauchamp would address it immediately. "Erik would answer a troubleshooting call even if he was driving. He'd say, 'Let me pull over and I'll fix your problem.' He was always there for us. He was so wonderful and the friendliest guy in person as well."
The Covid pandemic caused major changes in the production industry, requiring innovative solutions to overcome the challenges of remote editing. "So much great technology has been developed due to Covid; it's one silver lining of such a horrible pandemic and such economic upset. Remote workflows and remote review technologies all through production and post are having a big impact on the industry," says Spence.
It's an ongoing process and RPL is evolving with it because there are still hurdles that editors face. "One challenge is image fidelity. Since your monitor is displaying an emulation of an image, you're not seeing the actual image. This can cause you to not see something on your screen that you might have otherwise," says Good, who references a shot on TLOU where crew members were seen in the lower left of the frame.
"We didn't catch it in editing because they were super small and blended in with the color of the trees, and they didn't move. And the VFX team didn't see it because this wasn't a visual effect shot. Maybe if we had more resolution, we could have seen that. With remote editing, you're losing a little bit of the image because you're not seeing the actual image on your monitors. But if that's the price to pay for this kind of flexibility and better work/life balance, then that's a price worth paying," Good says.
Mendez agrees, "Since we stayed hybrid (with the option to edit at home or at the cutting rooms in L.A.) if a render took an hour to export, then I could drive home in that time and check it from there. That added more family time to my day-to-day life."
Spence sees the advantages of cloud-based remote editing systems and feels confident about their continuation as an industry norm. "People are getting used to having that additional work flexibility. RPL's system was definitely the most involved remote and flexible editing workflow that we've ever come up with and it always worked well for TLOU. That was a lot of fun," he concludes.
