Six Years of Methodology. Two Days to Automate.

By Guy Gelbart

In 2018, we started asking a different question about keyword research.

The standard approach treated keywords as a list to rank for. You pulled terms from a tool, sorted by volume, and built content around them. But we kept running into the same problem: two keywords could look completely different on the surface and still trigger the exact same search results.

That observation changed how we thought about the entire process.

If Google serves the same pages for different queries, Google has already determined those queries share the same user intent. The search engine has spent billions figuring out what users want. The SERP isn’t just a list of competitors – it’s a mirror reflecting user needs back at you.

We called it “search listening.” And for six years, we did it manually.

The Manual Process (And Why It Broke)

The methodology was straightforward in theory: gather all possible keywords, analyze the SERPs, and tag queries by shared intent. Group the ones that trigger the same results. Build content around those clusters.

In practice, it created two persistent problems.

Problem one: missing relevant queries. When you’re manually reviewing thousands of keywords, you miss connections. A query phrased differently, an industry synonym you didn’t recognize, a long-tail variation that shares intent with your core term – these slip through. And every missed connection is a missed opportunity to consolidate content authority.

Problem two: misattributing intent. The opposite error. You tag two queries as related when they’re not. You build one page targeting both. Google disagrees with your assessment. Now you have content cannibalization – two pages competing for the same intent, neither ranking well.

Both problems got worse at scale. A cybersecurity client came to us with thousands of technical terms. Industry jargon we didn’t know. Acronyms with multiple meanings. The manual process would have taken weeks, and the error rate would have been high.

So we built something.

What We Actually Built

Two days in Google AI Studio. That’s how long it took to turn six years of methodology into a working tool.

The core logic mirrors what we’d been doing manually, but without the gaps:

Step one: Ingest the keyword universe. Every relevant term for the client’s space – pulled from Ahrefs, Search Console, whatever sources apply.

Step two: Map each keyword to its SERP. For every keyword, we capture the top 10 ranking URLs. This is the raw signal – Google’s interpretation of what users want when they type that query.

Step three: Calculate similarity. For any two keywords, we measure how much their SERPs overlap. The metric is Jaccard similarity: shared URLs divided by total unique URLs. If keyword A and keyword B return seven of the same pages in their top 10, they almost certainly share intent.

Step four: Cluster using Complete Linkage. This is the part that prevents bad groupings. Standard clustering can “chain” keywords together – A connects to B, B connects to C, so A ends up grouped with C even if they’re unrelated. Complete Linkage only merges clusters when every keyword in one group is sufficiently similar to every keyword in the other. Tighter clusters. Fewer errors.

Optional: AI Hybrid mode. For sparse niches where SERP overlap is low, we added a semantic layer. The tool embeds both the keywords and their SERP content (titles, snippets) using Google’s text-embedding model. If two keywords have minimal SERP overlap but very high semantic similarity, they can still cluster – but only if the AI confidence is above 85%. Safety gates prevent hallucinated groupings.

The output: structured intent clusters, each with a representative keyword, supporting terms, and a clear signal for content strategy.

What Changed

The cybersecurity client had terminology we’d never encountered. Industry-specific acronyms. Technical phrases with precise meanings. Manually learning that landscape would have taken weeks of research before we could even begin clustering.

The tool handled it in hours.

It found synonym relationships we wouldn’t have caught. It grouped queries by intent even when the phrasing was completely different. It surfaced clusters we didn’t know existed—micro-intents within the broader topic space that deserved their own content.

We’ve since deployed the same approach across travel, fintech, web3, and insurance. B2B and B2C. The pattern holds: SERP overlap is a reliable intent signal regardless of industry.

The time savings are obvious. What used to take weeks now takes hours. But the accuracy improvement matters more. We’re not guessing at intent anymore. We’re reading Google’s own interpretation and building strategy around it.

The Deeper Point

This isn’t really a story about a tool. Tools are easy to build now – that’s the whole premise of vibe coding.

The story is about methodology. Six years of developing an approach, testing it across clients, refining the logic. The tool just operationalized what we already knew worked.

That’s the pattern worth paying attention to. Domain expertise doesn’t become obsolete when AI tools get powerful. It becomes more valuable. The people who’ve spent years understanding a problem are exactly the people who can build the right solution when the building becomes easy.

We didn’t automate keyword research. We automated our keyword research – the specific methodology we’d developed through thousands of hours of client work.

The insight that made it possible wasn’t technical. It was conceptual: the SERP is a signal layer. Google’s results aren’t random – they’re a reflection of how the search engine interprets user needs. Read them correctly, and you’re not guessing at intent. You’re observing it.

That insight took years to develop. The automation took two days.

What We’d Add Next

The current version solves the clustering problem. But there’s more signal in the SERP than just URL overlap.

Page types tell you what format users expect. If Google serves mostly comparison pages for a query, users want comparison content – not a product page, not a how-to guide. We’re building that layer in.

SERP features signal intent nuances. Featured snippets, video carousels, People Also Ask boxes—each one indicates something about what users want and how they want it delivered.

Position stability over time shows which intents are settled versus contested. If the same pages have ranked for years, the intent is clear and the competition is established. If rankings churn constantly, there’s an opportunity to define the space.

All of this is readable. All of it is a signal. The methodology keeps expanding because the SERP keeps telling us more than most people bother to listen to.

If you’re dealing with large-scale keyword research or struggling to map intent accurately, let’s talk. I share what we’re learning from real client work.