Redefining Promo and Sports Highlight Creation Using AI

By Adrish Bera On Nov 25, 2019

Promos are key for Media and Entertainment enterprises to create an audience pull. Media and Entertainment Marketing teams create promos to sell their content to a distributor/syndicator or to attract viewers towards their content on linear TV or OTT platforms. Promos are used for pre-produced entertainment content like movies and shows, news programs as well for live sports events. In the case of sports, they take on the form of what is known as highlights or highlight packages.

Since promos have a major impact on viewership, content creators spend a significant amount of time and resources in creating promos and managing promo operations. Let’s take a deeper dive to understand this in greater detail.

Promo Creation for General Entertainment Channel (GEC) Content

Creating promos for movies and TV shows is largely a creative task. There are typically 3 approaches to promo creation: 1) Script – shoot – produce 2) Edit and prepare promos from the main show’s available content 3) Combination of #1 and #2 – shoot and use available content.

For #2 and #3, significant time is spent in selecting the right content clips that bring the promo script alive. In the case of GECs, most promos are created for movies and daily soaps. For a given brief/rough script, promo producers and editors scrub through the content a number of times in search of interesting pieces. Once the selected clips are put in the editor’s edit bin, he/she trims them to create rough cuts that would be part of the final promo. Based on creative grammar, the clips are sequenced in a certain way to bring out the drama and captivate viewers’ attention. Next, suitable background music – either from the base content or from stock music libraries – is added to the promo. Finally, graphics, special effects and overlay text elements are used to finish the promo.

We studied this process carefully and realized that there was a lot of potential for an Artificial Intelligence (AI)/Machine Learning (ML) solution in this space to assist editors and producers. Our goal was to eliminate the manual heavy lifting, free up staff members to focus on more creative pursuits and drive unprecedented scale and speed. We embarked upon the journey of introducing AI-led Automation to the process of promo creation in various phases:

Base Camp Stage

In this phase, we targeted easy discovery and search of content for promo creation. Our AI engine runs a combination of third-party and homegrown AI engines to discover content and extract metadata in the form of keywords, transcripts, sounds, and tags. This metadata is synthesized and curated again by the AI engine to make the data accurate and contextual. The final harvested metadata is then indexed for an AI-enabled search.

This intelligent search capability enables an editor to search for scenes that he/she is looking for using natural language. Natural language search is key, as new-age users are accustomed to such features in Google search. For example, for the movie ‘Casino Royale’, an editor can simply search for ‘Bond comes out of the sea bare-chested’.

During our observation sessions, we also found that editors’ hands were always busy on their edit machines as they use many shortcuts for performing edit actions. So, we decided to give editors an extra hand – a voice search feature that responds to a cue word.

We also introduced a feature called Playlist, which analyzes content, identifies interesting theme clusters and collates interesting clips within these clusters. This enables editors to simply pick a relevant clip without having to explicitly search for it.

The AI-assisted workflow in the Basecamp phase is as follows:

Assets that an editor need to work on are pre-processed and made available for AI-enabled search
The editor can find interesting clips from the Playlists under different themes (like Romance, Car Chase, Blasts/Explosions)
The editor can also search for scenes he/she wants in simple English or by calling out search sentences
The editor creates a promo project and collates the clips from the playlists and search results
The project is exported to an edit software like Adobe Premiere Pro as an EDL (Edit Decision List) file
The cuts and final touches to the promo (rough cut) are applied within Premiere Pro
Background sounds, transitions, color correction, graphics, etc. are applied to the rough cut to shape the final version of the promo

This new workflow enhances efficiencies across the promo creation process significantly. In our trials, the time spent on searching for clips itself got reduced by 60-80%. 8 out of 10 times, the AI engine was able to generate results for all the search strings (nuanced, in natural language).

Mid-camp and Summit Stages

The idea for the next stages of AI development – which is still work in progress – is to automate the promo creation process further. In these phases, the AI engine is expected to match clips to a given promo script.

Promo creation is not just about identifying the ‘magic’ clips. Promo creators also follow certain creative grammar to make the final product impactful. For example, to make the story compelling, an editor needs to build and release tension at certain intervals. Potentially, AI algorithms based on learnings from similar projects can be used for the application of chosen ‘promo grammar’ (specific edit actions, etc.). Eventually, in later phases, AI can further automate the process by applying the right background score, bringing audio leveling and transitions within the content and even applying color corrections. Any manual interventions needed on top of machine-generated promos are captured by the system and used for continuous future Machine Learning.

Separating Reality from Myth

In 2016, a news item was published proclaiming that IBM Watson had automatically generated a trailer for the movie ‘Morgan’. This was after the AI engine had analyzed over 100 horror films and learned how to create such trailers. However, a human editor was still needed to stitch the scenes together to finish the trailer.

We believe that while efforts are underway to automate the promo creation process and make it almost ‘zero-touch’, we still have a long way to go. This is because promo creation involves deep human creativity and cognitive variance. In the meantime, the ability to assist humans with AI intervention is no small achievement and it can add tremendous value to the promo creation process.

Sports Highlight Creation

Apart from general entertainment, AI and ML play a vital role in achieving unprecedented efficiency in sports production, especially for highlight creation, boosting viewership and increasing ad monetization.

In fact, we have gotten better results by introducing automation for sports use cases as compared to GEC use cases – simply because the creative limitations are more defined in sports, and the scope of highlight creation is not as open as promo creation in GEC.

For example, creating a match highlight package needs to follow some set rules. A soccer highlight package will invariably have goals, shot at goals, long moves, red, yellow cards etc. A cricket highlight package will have sixes, fours, and wickets, etc. But a highlight package is often much more than that. The human editor artfully cuts a package that tells a compelling story by capturing the suspense, comedy, and drama of the game. If a batsman is beaten a couple of times before getting out, the editor shows all 3 balls – not just the last one. Other elements like tournament montages, pre-match ceremony, player entry, toss of coin, etc. also need to be included in highlights.

For spectators, excitement levels typically peak during the high points of a game. For instance, in football, this could be when a goal is scored or missed. Visual clues like a referee/umpire signal or text overlay on-screen also denote key events. Some providers have created engines that can tap these high points and attributes to create a simple highlight package.

But looking at the complexities and nuances of mainstream sporting action, we adopted a different approach. We have built custom engines to understand and catalog several aspects of the game including game objects, constructs, players, events and actions. This engine aids content discovery in near real-time and helps sports producers get to the most relevant content needed for a highlight package.

Our AI-generated fully automated highlights are created using learnable business rules that are built from past match highlight productions. These need to be improved over time as the engine learns what works better and what does not. These rules are based on comprehensive attributes/tags for each match segment. Much like a human editor, machines can also bring in AI-based audio smoothening, scene transitions between two segments, smooth commentary cuts, etc. In our trials with cricket, we found that the match highlight packages generated by our engines matched human-generated highlights by 85-90%. The remaining 10-15% that did not match were not necessarily bad edits – in fact, at times they were better than machine-cut highlights! In this use case too, the tool provides EDL exports in order for editors to do Quality Check (QC) and finishing, if required.

Today, producers need to swiftly publish highlight clips to platforms like Facebook, Twitter, and YouTube, for which they need game tagging and highlight generation to take place quickly. AI engines with both on-premise and Cloud hosting capabilities can help content creators achieve the necessary speed within the necessary timelines.

Clear Outcomes

In both the use cases described above, AI yields several concrete business benefits. These include enhanced efficiencies, lower Total Cost of Operations (TCOP) and increased monetization from content and ads. For both GEC and sports content, our goal is to help automate routine tasks in the creative process, enabling human resources to focus their intellect on far more creative pursuits. We are deeply focused on helping content creators achieve faster time-to-market, and improve user engagement through enhanced storytelling.