Prompt analysis: a new world to explore

Search analysis was born as a tactical tool for ranking on search engines. With the advent of chatbots and large language models, the raw material changes structurally – and with it some of its fundamental rules.

In the early days of SEO in Italy, the keyword analysis needed to define “Search Engine Positioning” strategies – organic ranking on search engines – was carried out without any dedicated tool, relying on common sense alone. Wordtracker had already existed since 1998 (it still does, by the way), but it offered data exclusively on English-language queries. For Italian there was practically nothing.

The first opening came with the advent of bid engines, ancient engines that hoped to solve the problem of result quality – and of an effective monetization model – by auctioning off the most prominent positions for specific keywords. The underlying idea was that no one in their right mind would pay a cost per click obtained on search results unrelated to the products or services on offer. The first global player was Goto.com, followed by the Italian Godado. History tells us that the real answer to the problem of result quality would arrive with Google, but the bid engines had the merit of making concrete data on users’ searches available for the first time. A structural limit remained, however: their user base was limited – Godado’s, we might say, paltry – so the data they offered was not particularly representative of the scenario. To have a truly broad and statistically significant sample, we had to wait for the birth of Google AdWords and its Keyword Tool. From that moment on the dataset became abundant… and with abundance, new problems emerged.

The digital confessional

For years, search analysis was perceived as an activity functional almost exclusively to SEM strategies, an acronym that stands for Search Engine Marketing in its broadest sense, namely the sum of SEO (Search Engine Optimization, organic ranking) and SEA (Search Engine Advertising, paid advertising on search engines). This limited its importance for many corporate stakeholders.

And yet the search box of an engine – and today we should add the interface of the various chatbots – is something far richer: a kind of confessional in which the user expresses their needs without censorship, a place where desires turn into measurable facts. Analyzing this data means gaining access to a true cross-section of society, an ocean of information that is valuable not only for search strategy, but for deeply understanding the potential audience and the target market.

The classification problem

As soon as the data became abundant, it was immediately clear that, to extract value from it, it had to be organized. Turning a shapeless mass of keywords into clean, ordered and intelligible datasets is precisely the goal of the activity that for years has been called Search Intelligence – and it is a far more complex activity than it seems at first glance, especially when working on thousands or tens of thousands of queries and aiming to articulate the classification across multiple levels.

Over time, different approaches have taken hold. The first is semantic: queries are grouped by meaning, with logics that can be nested – for example PC → Laptops → Screen size – or transversal, flagging all the keywords that contain a reference to a given attribute (processor type, RAM, storage space) regardless of the main cluster they belong to. In both cases, ambiguities soon emerge, and they must be managed with explicit, documented criteria. A query like “15-inch black laptop” can be assigned to the screen-size cluster or to the color one: the choice depends on what is most relevant to the client’s business and must be codified as a rule shared by the analysis team, not left to the discretion of the individual analyst. The same goes for undersized clusters: “15-inch retina mac laptop” might find a place in a “retina monitor” cluster or be absorbed into the “screen size” one, depending on the number of queries and the volumes present in the dataset.

The second approach is that of search intent: setting aside semantic aspects, it is possible to classify queries according to the type of intention that might drive the user. A query like “macbook deals” clearly suggests a transactional intent; “mac or pc, which is better” reasonably points to a consideration stage. In theory it is an elegant approach. In practice it presents structural limits that I have learned not to underestimate.

The idea that one can define “search personas” is as old as search analysis itself. To my mind, however, it is an approach as suggestive as it is inconclusive: in my experience it has almost never produced particularly useful operational guidance, only academic cases.

Many search strings are the expression of widely differing intents and, in some cases, entirely inscrutable ones. Classification by search intent is certainly useful, but it can be applied with a certain degree of certainty only to a limited portion of the dataset – typically to queries that contain elements unequivocally associated with a specific intent. The gray area, which often includes the most popular and highest-volume queries, remains for the most part impossible to disambiguate, limiting the operational guidance one might expect from this approach.

Richer prompts, sharper analysis

The establishment of LLMs as search interfaces has brought a marked change in the way users formulate their questions. Prompts are generally far more elaborate than classic keywords – probably because chatbots are perceived differently from traditional search engines. The natural-language understanding capability seems to lead users to personify the model and relate to it as they would to a person. The result is more natural language, with context-rich prompts that offer information capable of simplifying the classification activity.

A caveat: this simplification does not mean that, thanks to the advent of prompts, it has become easy to automate classification with the support of AI. Errors and genuine hallucinations are the order of the day even when the model is offered very detailed context and a substantial set of already-classified examples. Much progress has been made on this front, but there is still work to do, and today an integrated system based on back-and-forth between different AI models and ample doses of human in the loop is advisable.

Back to our prompts: their greater verbosity first of all simplifies some dataset-cleaning operations. Identifying and removing questions in a language other than the one under analysis, for example, becomes almost trivial. But the main advantage offered by these elaborate queries concerns precisely classification by search intent. Structured questions contain many more elements useful for the purpose and drastically reduce the disambiguation problems typical of more laconic queries (often those with the highest volumes). The percentage of the dataset classifiable by intent with good confidence grows significantly.

The same rules, better results

The rules of engagement for classification remain the same. In nested semantic classification, to manage ambiguities, it is necessary to define explicit conventions such as the prevalence criteria to use when a prompt could end up in two different clusters, and the absorption of undersized “species” clusters into those that represent their “genus”. Transversal classification applies, in the case of prompts too, only to part of the dataset, but with a marked reduction in ambiguities and doubtful cases.

What changes, in fact, is the quality of the signal. Prompts offer more indications about the informational need expressed by the user and the search intent that underlies it. This makes it possible to work with greater precision on three fronts that in my experience are closely connected: the definition of the competitive scenario, the content strategy aimed at closing the content gap, and – something that is often underestimated – the structure of the individual pieces of content. Because content must not only intercept a search intent: it must satisfy it. And to do so, you need to know precisely what the user is looking for, not just intuit it.