FirstBranch.ai: Conversational Intelligence for Congress

I just finished building out a conversational intelligence tool for Congress. If you don’t know what this type of tool does is, here’s a quick summary. You have many videos, transcribe them, and build a search engine on the context of those videos (among other tooling around them). You can then use the videos to answer questions. It is commonly used in sales, where account executives share the pitches that work amongst each other, and product managers watch customer calls to discover how to improve and differentiate the product.

At a high level overview, a conversational intelligence tool catalogues videos, extracts transcripts, indexes the content of those videos, ideally chunked so that specific fragments can be referenced, and provides analytics on them. This unique open dataset of hearings is an excellent playing ground for trying out techniques such as large-scale LLM data processing and summarization, unsupervised clustering algorithms, and search based on document embeddings.

My goal with this small project was to improve over the existing Congress website in 3 areas: visualization, search, and viewer UX. And this is precisely what I did with Congress hearings. I processed all of the hearings over the last 2 years, made them searchable, and provided a viewer experience focused around finding the most interesting moments of a hearing, rather than sifting through hours of video. This is a quick technical write-up on my learnings and reflections.

Congressional Hearings: the raw material

Hearings are the raw material - everything the website does is to help you discover hearings and surface the most insightful moments. Because of that, the data pipeline to ingest hearings is of utmost importance. This pipeline includes:

• Listing extraction: Extracting metadata about the latest hearings using the congress.gov API, including paths to the video recordings.
• Transcript extraction: Converting those video URLs into transcripts - there’s two types of URLs, YouTube, and Senate-isvp (hosted by the Senate). Fortunately, the transcripts come with timestamps.
• Summarization: One of the most interesting steps is turning the transcripts of multi-hour meetings into one-page summaries. There are a few highlights:
  • One-shot learning: LLMs are non-deterministic by nature, but a product like this one requires a deterministic output across hearings to be useful. Fortunately Sonnet 5 is quite good at following instructions, however, I found it highly valuable to apply one of the old trusty LLM prompting techniques, few-shot learning a.k.a. providing an example. I carefully crafted a prompt and an example summary I was happy with based on a hearing I actually listened to end to end, on price transparency, a topic I’m interested in, to ground the model on important conventions. This provided grounding in terms of relevant sections and structure (every speaker gets a timestamp of their first moment of speech) as well as in terms of taste about key moments and how long sections should be. While a prompt leaves room for interpretation, the example provided consistency across summaries.
  • Citations: While I ended up not using the Claude Citations API, I made the model provide a [hh:mm:ss] timestamp for each of its claims. This reduces hallucinations and allows the user to check the statement in detail. Providing links to specific moments in the video is the key feature of the app: you discover interesting moments skimming the transcript and then watch those moments, skipping the fluff.
  • Verification: Also an important component for achieving consistent results. While LLM verification was too pricy (more on costs later) programmatic verification offered lots of room for checks, including i) ensuring timestamps are in the [hh:mm:ss] format and that these are within the bounds of the video, and ii) ensuring that all of the key sections are present in the summary.
  • Costs: The driver of the project costs is certainly Sonnet 5. Measured from my Anthropic usage data - I used claude -p (headless) - I spent ~114M in / ~7.6M out tokens. At API rates, this would be $800+. I managed to do it with my Claude Max subscription, over multiple sessions, and $200 in expiring usage credits. Embeddings for the same tokens, at $0.12/1M don’t come close.
• Polish: For every hearing, I also provide a thumbnail (for the purposes of search results, link sharing, and video previews) and metadata straight from Congress. I also provide categorizations derived from clustering, and grouped into manually labeled themes. More on how these are produced below, alongside search.

Chunk Embeddings: the foundation of map and search

Two of the most fun features to build come from embeddings. I was originally inspired by this post that mapped all regulations in Australia. It turns out that you can, with embeddings, turn a corpus of documents into a map. You can then, look at points that are close together in this map (or maybe in the 2048-dimensional space that you transform into a 2D map 😉) and you’ll uncover clusters of similar items, like meetings discussing marking up bills about veteran affairs, or those discussing Congressional rules, even if these are made by different committees.

Just like you can use embeddings to make a map, you can also use them to search. I believe my search engine is better than https://www.congress.gov/search. But even if it isn’t, at least it looks cooler on a map, and it links you to the exact second the hearing starts (which would be anything between a minute and an hour into the recording).

The pipeline for mapping and searching includes:

• Summarization: from the “hearings” section above, as a prerequisite
• Embedding: I had to make a few decisions in this step, to make the embeddings as high quality as possible, including:

  • Choice of embedding for search and clustering: the choice of embeddings and chunking strategy in search is widely discussed and I will not pretend to be an expert. But I did consider different alternatives for my project: mean-pooled chunks, summary embeddings, and the eventual winner: contextualized summary embeddings. Voyage AI has developed a context model that embeds chunks in the context of their surrounding document. Chunks, however, can be noisy, and, like I saw via trial and error, it can result in matching topics that are mentioned in passing. Summaries alone are not able to capture the nuance of highly specific queries unless it is explicitly referenced. However, combining both into the same document, the contextualized embedding of the summary also carries some of the nuance of the transcript and seems to achieve best results.

  • Tokenization optimization: Preprocessing is essential in ML and AI (garbage in, garbage out) and so it is here. The embeddings API I used is called Voyage AI and has a limit of 32k tokens per document. Initially, Claude was estimating tokens as 1.4x words which turned out to be highly inaccurate by + and - 25% depending on the video platform. The root cause was that Senate videos were improperly cased and needed to be preprocessed to be sentence cased, since tokens group frequently occurring characters together and capital case is, in general, more infrequent.

    

    Demanding answers when Fable tried to ignore me had a significant impact - it is important to stay at the wheel on important technical components of the system. Switching to measuring tokens with a proper tokenizer and fixing the casing allowed me to ensure that 80.7% of hearings fit in a single document (32k tokens) while initially this was just 48.2%.

    

• Clustering: I followed this blog post closely and did not pursue further research. Most of my efforts were grid searching for the right combination of hyperparameters (there were 4) and visually inspecting the result to find the most aesthetically pleasing map. I had Fable label the initial clusters and group them together into themes, and I manually renamed and merged based on that initial draft to my taste. The categories are what you can see in the legend of the map if you visit the website. If I had more time, I would have spent more time understanding the different algorithms available, but I already felt technically satisfied.
• Search: There is not much to my implementation of search. I use Voyage AI to obtain an embedding (one of the only two API calls my web makes, the other being for video rendering, otherwise it could be fully static). Then I run cosine similarity with DuckDB, which can handle concurrent queries. There’s some questions about scalability and continuously ingesting data, and it might be good to consider alternative data stores. DuckDB is good for a quick prototype and handles my current corpus size well.

Delighting users: the design

I also spent a meaningful amount of time refining the UX. Here’s some of the design decisions that I made:

• Theatre view: I followed Gong’s conversational intelligence product for salespeople whose key job is to analyze meetings, transcribe them, and highlight important moments and actions to take (as well as making those easily shareable with your colleagues). They strike a good balance of making the video the star of the page, as big as possible and rendered in its original aspect ratio, while also providing their intelligence in the form of relevant metadata about the video, a transcript you can search for specific things, and the highlights. You could think of the hearing detail page as “Gong for Congress”.
• Map hearing previews: I invested some time in making the hearing preview experience in the map bearable (if not delightful) by adjusting several finicky aspects about tooltips, including:

  • Placement: they cannot go off the screen, nor can they be too close to the pointer because that prevents you from visiting neighboring nodes.

  • Animation: there are 2 key levers. i) the time it takes for a tooltip to replace an existing one when hovering on a new node. If it is too quick, hovering over other nodes while trying to click somewhere in the tooltip will make it disappear. ii) the time it takes for a tooltip to disappear overall.

  • Presentation: I wanted to make the tooltip engaging and show thumbnails consistent with the meeting.

    

  • There’s plenty of room for improvement, from showing category badges, to cleaning up some of the text, to adding a play icon on the thumbnail, and maybe even embed a video player if you’re adventurous.

• Tiny wins, quality of life features that make the app a delight to use:
  • Timestamp lags: while the transcripts had timestamps, those were consistently time-shifted. This is essential to support the citation feature and any transcript timestamp.
  • Begins at: a feature that ensures that when you click on a video it starts playing when the committee begins, which is an indeterminate amount of minutes and up to an hour from the recording start
  • Search emphasis: any search query or filter query shows specific search results that you can play and highlights them on the map so that you can explore their neighbors
  • Filtering and sorting: the map view lets you pick date ranges, select specific themes and clusters to filter for, etc. I iterated on them to ensure that the date picker was user-friendly (no calendar range selection, unless you really want to) and the legend was interactive.

Growth: SEO and link sharing

While I did this for fun, I’d love for this to be valuable. The more people it reaches, the more it can help others. And growth is also an interesting problem to think about. I made a few things to help my app reach more people, directly or indirectly. Any reference, or even an LLM influenced by this product is something I’d be proud of and want to encourage.

• Static site and pre-rendering: all of the pages in the app can be referenced in a sitemap and pre-rendered with full transcript and summary content. This ensures that LLMs and search engines will pick it up and provide me with traffic. Most of my traffic will probably come from long-tail searches that don’t have a good match outside of hearing transcripts or summaries. Adding the sitemap to the Google Search Console and linking the website in a few places will boost its PageRank and help me get some impressions and clicks.
• Social link support: I leveraged Opengraph to ensure that my page is shareable across multiple platforms and generates good previews. Here’s the example Opengraph score of a hearing. This ensures that if you send a hearing link it shows the right preview title and thumbnail. You can verify this by pasting one of the hearing links in Notion and hovering over it.
• Share button: I provide a share button that links you to specific moments in the video, and, browser permitting, autoplays on click.

The things I left for a later day

The only other thing that I wanted to explore was building an assistant that could leverage RAG to point to specific chunks in the video. Exposing this online would also have required authentication, so that I can rate limit users and gate access to whatever model I provide. Or I could have gone the open source route, but that would’ve required standing up infrastructure outside of my closed-source monorepo which has some deployment tooling set up. So in a future day I could:

• Implement auth, for example via Firebase
• Implement rate-limits
• Build an assistant, and have it use a tool to reference chunk embeddings all across the hearing universe to provide contextualized replies with links to specific video moments
• Open source the data and the codebase so that you can bring your own key and talk with the assistant
• Set up incremental ingestion of new hearings and backfill data from 2024 and earlier

I hope you liked the post, and that you’re inspired. If that’s the case, go build!