AI Agents, Standard

How Social Department built culture-aware video discovery with QuickSearch

Symbolic diagram: How Social Department built culture-aware video discovery with QuickSearch

Introduction to culture-aware video discovery systems

Social Department’s QuickSearch turns editorial questions into executable video retrieval pipelines for culture-aware video discovery. Instead of asking marketers to describe the clip they want, it offers four actions: Shareable Scenes, Quotable Quotes, Memeable Moments, and Trend Align. Each applies its own retrieval strategy, editorial criteria, and ranking model to propose material from a video library.

The central problem is the gap between identifying what footage contains and judging why an audience might respond to it. Finding a visible object is a different task from finding a scene that could travel socially. QuickSearch addresses that gap by combining structured understanding of a content library with external cultural signals and purpose-specific retrieval.

The architecture is described in an AWS post co-written with Social Department. It reports that entertainment marketing teams manage tens of thousands of assets across studios, networks, and streaming applications. At that scale, relying on a person’s familiarity with each show creates a discovery bottleneck. The authors describe earlier workflows in which someone with deep knowledge of a title could spend a week watching footage to answer an editorial brief.

The account supports a concrete architectural explanation, but not a measured claim that QuickSearch improves campaign performance. The supplied evidence contains no retrieval accuracy benchmark, controlled comparison, or quantified productivity result. Its significance is the attempt to encode editorial intent into repeatable retrieval operations while keeping people responsible for the final publishing decision.

Social Department owns the pipeline architecture, retrieval system, and editorial criteria; TrackIt implemented the design. The documented AWS components include Amazon Bedrock for model access and Amazon OpenSearch Serverless for retrieval. The source explicitly limits its coverage to the AWS portions of the system. Social Department’s broader media backbone, discovery engine, and evaluation program are described only as needed to explain that architecture.

Challenges in traditional video search methods

Symbolic diagram illustrating: Challenges in traditional video search methods

Semantic search can retrieve material that matches a description. The source illustrates this with a character holding a coffee mug: a system may find matching objects across many episodes, yet that does not identify which moment deserves a campaign. The difficult questions concern audience response, quotability, or a frame’s potential as a meme.

The limitation is therefore partly about who supplies the intent. A marketer who already knows the right scene can describe it. Someone exploring an unfamiliar catalog may need the system to propose candidates before they can specify what they want. In the authors’ account, a content-focused search layer leaves that editorial work with the person making the query.

Two missing inputs matter. One is an editorial definition of what counts as shareable, quotable, or memeable for a particular library. The other is current information about conversations outside that library. Understanding the footage cannot, by itself, reveal what audiences are discussing that morning.

Those inputs also change at different rates. Library understanding needs to change when content is added, whereas external trends can change hourly. Social Department consequently separates content understanding from trend acquisition and joins their outputs when a user chooses a QuickSearch action. This is the core design decision behind its culture-aware video discovery approach.

Making the library searchable still requires substantial preparation. Video uploads enter a per-team Amazon S3 bucket, and an event starts an AWS Step Functions workflow coordinating roughly two dozen Python AWS Lambda functions. The pipeline resolves metadata, transcodes footage, detects shots, and merges or splits them into content-aware ranges of roughly 10–20 seconds.

TwelveLabs Marengo 3.0 on Amazon Bedrock generates multimodal embeddings from those ranges. These are representations used for similarity retrieval across visual, audio, and transcript dimensions. TwelveLabs Pegasus 1.2 separately produces scene descriptions, emotions, themes, dialogue type, and narrative position. Transcription, on-screen text detection, visual labels, and character grounding add further searchable context.

These operations have distinct input requirements. Marengo receives start and end times without requiring file splitting on that path. Pegasus receives pre-cut files, with padding around each range to preserve transitions. The pipeline ultimately indexes embeddings and descriptive fields into OpenSearch Serverless. The practical implication is that editorial discovery depends on multiple representations of a scene, rather than one universal description of the video.

Overview of QuickSearch functionality

Symbolic diagram illustrating: Overview of QuickSearch functionality

QuickSearch presents a simple selection to the marketer, but each selection initiates a different retrieval process. Its four actions make the editorial objective explicit before the system searches.

ActionDocumented retrieval objectiveImportant qualification
Shareable ScenesFind substantial, self-contained scenes, including confrontations, climaxes, and set pieces.Favors developed sequences over brief reactions.
Quotable QuotesFind punchy dialogue in short clips.Gives extra weight to previously circulated lines when a popularity signal is available.
Memeable MomentsSelect individual frames suitable for memes and generate captions grounded in those frames.Uses dedicated frame extraction and scoring rather than simply taking a thumbnail.
Trend AlignFind library moments connected to current trends.Requires translating cultural topics into searchable audiovisual descriptions.

The actions supplement natural-language search and image-based search. A reference image can return visually similar library content, while natural-language queries remain available for needs outside the four predefined actions. QuickSearch therefore adds an editorial entry point without eliminating descriptive search.

Once selected, an action creates a job record in Amazon DynamoDB and places a message in Amazon SQS. A worker Lambda function executes the corresponding pipeline asynchronously. Transient failures retry through SQS; errors classified as non-retriable go directly to a dead-letter queue.

Each action expands into retrieval queries shaped by its editorial criteria. Trend Align illustrates why this is more than a shortcut around typing. A cultural topic is not necessarily a description of anything visible or audible in a video. Claude models on Amazon Bedrock translate that topic into queries the index can match. The source identifies this translation as an area containing substantial proprietary work, so it does not provide enough detail to reproduce the editorial logic.

Retrieval combines several routes. Marengo embeds each query once, accepting text, an image, or a composed text-plus-image query. The search layer uses six k-nearest-neighbor channels—visual, audio, transcript, on-screen text, scene description, and labels—alongside seven BM25 text channels. The former support similarity search; the latter provide text retrieval. Query-specific weights and routing adjust their contributions, with dialogue queries emphasizing transcript and emotion channels and visual queries suppressing them.

Reciprocal Rank Fusion combines the channels using result positions rather than raw similarity scores. That matters because scores from different retrieval methods do not necessarily share a comparable scale. Top candidates then pass to the Amazon Rerank model through the Amazon Bedrock Rerank API. The supplied source ends during its explanation of this stage, leaving later ranking or delivery details undocumented here.

Operational details also constrain deployment. For Marengo asynchronous invocation, input and output S3 storage must share the model endpoint’s Region. The described pipeline stages video in US East (N. Virginia), runs the embedding job there, and reads the output back while the rest of the infrastructure remains in its home Region. This is a documented architectural dependency, not evidence of universal model availability.

Responsible AI practices in content retrieval

Social Department’s stated policy places AI in a proposing role and leaves decisions to people. In the documented workflow, a person on the customer’s team approves every post before publication. That human checkpoint is especially relevant when the system is making editorial suggestions rather than returning literal matches to a narrowly defined query.

The system also checks generated output against source material. The authors give a specific production example: a proposed quote is discarded if it does not occur in the episode transcript. This connects a generated suggestion to an inspectable reference and prevents that particular unsupported output from progressing. It does not establish that every editorial judgment or generated element is correct.

Amazon Bedrock Guardrails helps screen customer search text for prompt attacks. The source also states that customer data stays within the AWS data boundary, is not shared with model providers, and is not used to train the underlying models. These are claims in the AWS and Social Department account; the supplied evidence does not include an independent security assessment.

Trend acquisition adds an editorial control before material reaches customers. Claude Haiku normalizes raw trends into Social Department’s category schema, standardizes descriptions, removes near-duplicates, and limits the number retained in each category. In that pass, an editorial gate rejects trends primarily about death, violence, self-harm, natural disasters, politics, protests, crime, or public-health crises. Rejected trends are remembered to avoid repeated screening on subsequent fetches.

That gate defines the scope of the recommendations. A highly popular topic can be excluded because Social Department does not consider it suitable for its marketing feed. As an editorial interpretation, this makes the product’s notion of relevance conditional: a connection must fit both the content library and the system’s marketing criteria. Popularity alone is insufficient.

The documented controls address different failure modes: transcript checks constrain quote generation, search screening addresses prompt attacks, trend filtering narrows offered topics, and human approval governs publication. The source does not provide error rates or evaluation results for these controls. Their presence explains the intended oversight model without demonstrating how reliably it performs across titles or audiences.

Importance of trend signals in video marketing

Trend Align depends on information acquired outside the library. The architecture keeps that acquisition separate from content processing so cultural context can refresh without waiting for new footage. At request time, the job layer reconciles the available trend information with indexed video content.

The source describes two acquisition lanes with different schedules. A daily lane uses an Amazon EventBridge cron to fetch once per region per day, timed to the region’s local morning. A second lane, Heat Seeker, retrieves early-stage trends ordered by momentum only when a user explicitly requests it. It reuses stored information when that information is recent enough.

This qualifies the broader description of continuous trend acquisition. The system maintains an ongoing trend pipeline, but the documented feeds do not all refresh continuously in real time. The daily lane is scheduled; the heat lane is driven by demand and cache freshness. The source does not specify a freshness interval for that cache.

The authors explain the split as a cost decision. Repeated scheduled fetching would incur work for every region regardless of customer activity. A demand-driven fetch with a time-to-live avoids fresh acquisition on a quiet day. The trade-off is an explicit relationship between acquisition frequency, user demand, and the age of available trend information; the evidence provides no numerical cost comparison.

Regional relevance is also built into acquisition. The documented feeds cover worldwide trends, the United States, the United Kingdom, and Australia. Each is discovered separately on its own schedule rather than produced by filtering a global list. That distinction matters because region-specific discovery can supply different topics for the same day, giving the retrieval pipeline different starting points.

Interpreted trends are stored using stable fingerprints. A recurring trend reuses its row and cached derivations, while trends that disappear age out and are eventually retired. Refresh runs acquire a claim to prevent overlap. Quotable Quotes uses another, narrower external signal to boost dialogue that has already circulated; that input is optional, so its absence removes the boost rather than failing the job.

Together, these details show what culture awareness means in this implementation: editorial criteria guide the search, external signals supply changing context, and multimodal retrieval connects those inputs to footage. They do not prove that a selected scene will become popular. The substantive development is a system for proposing culturally relevant candidates from a large library, with the final judgment still assigned to the marketing team.