Improving search relevance

So lets say you wanna find freelancers for an upcoming job. Where would you start?

Customers need a specific type of freelancer, for instance, a wordpress theme developer – a role. However, when they run a search, they’ll only enter a skill, like wordpress.

They expect to see wordpress theme developers in the results. Now sure, they’ll be in the results… Somewhere, along with every other type of wordpress freelancer.

All of this came to light when I started digging through all the search tickets and comments. A lot of customers were mentioning the poor search relevance, and they didn’t think their query was the issue, they felt it was our algorithms that weren’t working correctly. Which makes sense, currently there’s no guidance on what a good query is.

How can upwork guide customers to more relevant results?

It starts with figuring out what is an ideal query. We know from the logs the most common search queries only contain one skill…

I took that skill and looked at all the keywords customers entered after that skill.

There were more roles then this. These we just the most common.

These turned out to be types of roles, rather than multiple skills, which is a significant differentiator. For instance, there is a big difference between wordpress developers and designers.

There were more specialties then this, these were just examples.

I repeated this process with each of the roles. I Doing this I found queries for more specific types of roles, a specialty role.

This role structure was consistent, when looking at multiple types of jobs across the marketplace.

The nice thing about this, is it not only guides clients to better results. It teaches them to be more specific when searching.

The results are more relevant, but how can we make that clearer?

Upworks search algorithm weights the ranking of freelancers on a bunch of attributes. The weight is pretty heavy on the title. So the closer the title matches their query, the better, the more relevant they look

But what about freelancers work experience?

We get a lot of feedback about the search ranking. Customers don’t understand why one freelancer is at the top with little experience, and another was at the bottom with lots of experience.

Job success is feedback that is more predictive of future outcome.

So whats going on? The algorithm is ranking freelancers with relevant experience higher. Tiles only display overall experience, which is causing the ranking confusion.

Can’t we just display the hidden relevant experience to clear things up?

The problem is, it doesn’t exist. The algorithm ranks by it, sure. All its doing is spitting out the rank, a number out of 10. So the first step was to make one that gave us all the relevant data.

Before running to the data science team with the request. I needed to confirm with clients if they would want to see this relevant data.

I mocked up the example tiles then put them in a survey. The goal was to find out which information would customers prefer seeing when searching. Here are the results.

More than half (48) of the respondents preferred to see the number of relevant jobs & amount earned. Over a list of skills.

Once confirmed it was valuable to customers, I started designing. I wanted to keep the design direct and straightforward. Clicking on the number of jobs opened the profile, with a filtered view of the work history. Showing only those relevant jobs.

What was the outcome of these two features?

The results were pretty great, moving every metric we have. That was the second design that helped us align on a more structured marketplace that I’m working on currently.