Interview with Dan Sharp of Screaming Frog: On the Future of SEO (2020) Plus: Marketing Tips for Small Businesses

Today we’ll be speaking with Dan Sharp, co-founder and director of Screaming Frog.

As a UK SEO Agency, Screaming Frog was recently recognised at the UK Search Awards, winning Best SEO Campaign.

More than a successful agency, Screaming Frog is also the name of the widely used SEO spider software – a tool that has become a must-have for all SEO professionals as the leading onsite technical SEO auditing software.

Used by top SEO’s and marketers from Neil Patel to Rand Fishkin, the Screaming Frog SEO Spider is trusted by thousands of agencies and businesses around the world.

With a unique name – based creatively on a frog that stood up for itself after being cornered by 2 cats in Dan’s own garden, it comes as no surprise that creativity and the ability to adapt has been a hallmark of both Dan and his agency’s success over the years.

As the Screaming Frog agency has grown to now accommodate a team of 40, Dan was the ideal candidate to speak to in our ongoing series to uncover the secrets from the best and brightest marketing minds in the industry.

In our interview, Dan dives deep into the world of SEO. 

Some of the topics we explore include:

• The challenges he faced in growing his agency and SEO software
• His thoughts on the state of SEO in 2019
• His predictions for SEO in the future
• His marketing tips for small businesses.
• PLUS: His frustration at what Google is doing to the SEO industry right now.
• And much more.

Dan, thank you so much for taking the time to speak to Search It Local today.

Now, there’s a lot of information online about Screaming Frog, but our audience would love to know more about your story. What obstacles and challenges did Dan Sharp have to overcome to be where he is today?

There’s been so many along the way. We’re a search marketing agency, and we also develop SEO software which is widely used in the industry, and each side of the business brings different challenges. Also, the challenges have changed for us and me personally so much over the last 9 years.

I think the most significant challenge for me personally was the sheer amount of time, energy and commitment required to start up and run Screaming Frog, in the early years in particular. 

We had a small inexperienced team and I would work all day and all evening regularly for a very long time, mixing between training the team, consulting for clients, new business, helping build our tools, and running technical support.

People often think that running your own business gives you more time, or you are more in control of it. That’s often not the case at the outset. 

I didn’t have a holiday for years and rarely ever took a break. I am not complaining, I loved it, I am someone who can’t turn off anyway, but it was kinda exhausting. 

Luckily today, I have an amazing team and work/life balance has improved.

You mentioned that it’s been 9 years since Screaming Frog was founded. If you could do it all over again, is there anything you would do differently?

There’s lots of things I would do differently, but at the same time, it’s been an incredible journey, so fundamentally I wouldn’t do anything that differently either. It would be a shame to change anything, because everything has helped us become who we are today.

Although, I’d probably not use UK date format (dd/mm/yyyy) for our licensing expiry in our software again! This caused a year of confusion for American users who have a mm/dd/yy date format and would get in touch saying their expiry date was incorrect. Whoops.

Speaking of the software that you’ve become so well known for helping create. Is Screaming Frog SEO Spider only for marketers or would small business owners benefit from using this tool too?

I think anyone with a website could benefit from using the tool, but it is built for professional SEOs who know what they are doing. It’s not really aimed at those without experience.

It doesn’t tell you how to do SEO, it provides data to make informed SEO decisions. So, it requires someone with expertise to make the most of it and might be confusing for those without any base knowledge of SEO at all.

Screaming Frog SEO Spider tool – User Interface

Assuming a small business owner did have some SEO experience, in what ways would you then recommend a business owner use your tool?

There are so many ways our SEO Spider tool can be used. You can crawl your website like a search engine and see what pages it can index as well as review key elements that can impact SEO.

At a basic level you can find issues like broken links and redirects, or see your page titles, meta descriptions and headings. 

At a more advanced level (that probably wouldn’t be appropriate for many business owners), it can help you perform full technical SEO audits, content audits and many other specific tasks like auditing redirects in a site migration, hreflang implementation, and structured data validation.

You’ve touched on a really interesting point about the skill level required to implement technical SEO changes on your own website. 

Do you think a small business owner can master SEO or is an agency required to truly scale up at speed?

I think anyone can master SEO if they have the time, energy and attitude to learn. But small business owners generally don’t have that time, so it’s unrealistic. 

That said, I believe owners can learn the fundamentals, which is generally enough to give them an edge on the competition or at least, help them choose the right SEO consultant or agency to work with.

Let’s assume a business owner wants to learn more about SEO, wants to optimise their own website to improve their search engine rankings.

What are some simple SEO changes a small business can make to boost their Google rankings?

I think getting the basics right is important. Having a website that’s accessible to search engines, is compatible across all devices, and has genuinely useful content that meets the need and intent of the searcher is fundamental.

After that, carrying out keyword and customer research to better understand the types of phrases they use to find your products and services and then optimising landing pages, through improved internal linking, page titles, descriptions and on-page content to target those appropriately goes a long way.

Often a major consideration for small business is local SEO. Ensuring they have a Google My Business listing and have local landing pages covering relevant local queries can make a big impact.

Fantastic advice. We feel like there are countless businesses who feel overwhelmed by the thought of SEO and fail to take advantage of local search ranking signals that don’t break the bank or require overly technical knowledge, but still drive meaningful business outcomes.

As a director at Screaming Frog we’d love to get your thoughts on this next question, how much would you recommend a small business spend on marketing, and of that marketing budget how much should be spent on SEO?

This really depends on the business, where their customers are and the situation they are in, I don’t believe there’s a hard and fast rule.

But I do think you get what you pay for, so SEO for peanuts will give you peanuts.

You can work out the proportion of business from organic as a gauge for marketing split, but this doesn’t stack up in all situations obviously.

We agree. ‘You get what you pay for’ is particularly appropriate advice when it comes to digital marketing.

What advice would you have for business owners who are on the fence about partnering with an SEO agency?

I don’t think SEO agencies are right for every business. So I’d recommend they speak to prospective agencies and get a better understanding how they’d work together, and speak to the agencies current clients as well.

SEO isn’t something you can just give to another party to perform and forget about it, the best campaigns and results are collaborative efforts where the business and agency are aligned towards the same goal.

You’re obviously a highly skilled and experienced SEO professional, with your Screaming Frog SEO Spider tool reflecting that. So we want to get some first-hand insights about the optimisation process that my audience can take with them.

Google has shifted to prioritise quality, intent-matched content over keyword content. With this in mind, would you still recommend business owners and marketers pay attention to on-page factors like Title Tags and Meta Descriptions?

Absolutely. They still play a role in scoring, and enticing users to click on your search result over the competition.

You can have the best content matching the intent of the searcher, but if you have a terrible, irrelevant page title and description then it will be weighted less in scoring and may impact CTR when it appears in the SERPs.

Sticking with the theme of Google’s evolving approach to rankings. How have you changed your approach to SEO over the years?

Fundamentally it’s still the same, but our approach has matured and is far more sophisticated as search engine algorithms themselves have evolved.

SEO audits are more comprehensive for example, including far more UX elements, while link building is harder and content and digital PR campaigns need to be exceptional to stand out.

You’ve preempted our next question here by saying that ‘link building is harder’.

Do you see backlinks as being as valuable as they once were, and where do you see the importance of backlinks changing over the next five years?

Yes, links are still the backbone of scoring alongside content relevancy. Google still uses them to calculate the popularity, reputation and relevance of a website to a query.

You could argue they are more important than ever, considering how the link graph has changed, and how it’s harder to get links today. Without them, you won’t rank for competitive queries.

Conversely, links only go so far these days. Links help put you in the running to be considered for queries, Google then evaluates the results based upon user satisfaction data they can measure and improves its results to show the results delivering the best answers.

So if you’re a terrible answer to a query, or perhaps just not as good as the competition, you’re swiftly be demoted (even if you have better links).

I’m curious to know, particularly given your experience in the SEO industry, what would you change about Google’s ranking algorithm?

For many years I have said Google should reduce the reliance on links, and they have to an extent done this. I’d personally consider brand signals (such as brand search volume) more within the algorithm, as I feel this is probably more of a valid signal of popularity today.

For a long time I wanted Google to stop penalising websites for unnatural links (through old iterations of Penguin), and instead simply ignore the value of links. They did this a couple of years ago with Penguin 4 to their credit.

There’s plenty of other improvements required though, with the latest core algorithm update. A couple of small frustrations with the SERPs…

• We are seeing quite a few results with the same domain 3 times in a row in a single SERP. There needs to be greater diversification, and it feels like something is amiss currently.
• Google seems to be ranking low value add ‘listing’ type websites for plurals. For example, in our industry if you search for ‘SEO agencies’, or ‘PPC agencies’ in the UK, you’ll see some very poor websites that require a fee to be listed and ranked as a provider on their page. Obviously this ranking has nothing to do with quality, and these sites rely on those websites linking back to them to rank. Surely Google can do better today.

Perhaps wishful thinking, but I’d also like to see Google play fairer, they come into industries like ‘flights’ with a new product and immediately put themselves at the top of the SERPs. 

This should be challenged, as they are not actually the best user experience, so is it really the best results for users?

Do you believe SEO has become more accessible or more challenging?

I think it’s more challenging, as it’s not as simple as having content and then throwing lots of links at it.

We are much closer today to actually ‘needing to be the best answer’ to be able to achieve first place ranking (for example). 

Historically, you didn’t need to be the best answer, you could spam to the top.

Technology such as JavaScript is now also being used more (due to JS frameworks like React, Angular etc) and SPAs/PWAs, so this brings more challenges to SEOs who need to have an understanding of slightly more advanced concepts like JavaScript crawling and indexing.

With your view that SEO has become more challenging and requiring a more creative and time-sensitive approach, do you believe businesses should focus all their time and resources on one marketing channel, like SEO or Facebook ads, or adopt an omni-channel approach?

I’d weight and focus investment and resources on the relevant channels where the customers are, which will obviously depend on the business.

If customers are across many channels, then I’d test to see which had the most impact and best ROI and focus from there.

How do you see SEO changing over the next 5 years?

I don’t think we’ll see any dramatic shifts over the next 5-years, I just think we’ll see more of the same evolution. 

Mobile and voice will continue to rise and SEO will become more complex. We might see more of a shift into visual based SEO as well, which would be very cool. 

We’ll probably see Google try and kill a few more industries as well by creating products in that area.

Your Screaming Frog SEO Spider is widely used in our industry, making you a thought leader.

But who do you turn to for advice and inspiration, and are there any resources you would recommend that our audience use to inspire them?

I turn to my team for advice and inspiration. But there’s also plenty of smart and inspiring SEOs in the industry, and I follow a whole bunch of them on Twitter.

For news, I’d recommend Search Engine Land and Search Engine Roundtable.

What does the future hold for Screaming Frog, both the agency and the software?

More of the same. For software, we enjoy building innovative new features for the industry and making our customers lives easier. 

We were the first to launch JavaScript rendering, and used Chromium for rendering, literally years before anyone else. We recently released version 11 and this has structured data validation, which is another first, outside of Google’s structured data testing tool. We’ve got a lot more to come, which I am still (after 10 years of building it) very excited about.

For the agency, we’re growing. There’s 40 of us now in total, and we get to work with some amazing brands on incredible projects. I am generally not a fan of industry awards, as I think they have inherent problems – but that said, we were recognised for the Best SEO campaign last year at the UK Search Awards, which was rather nice.

Dan, we always end each interview with the same question, ‘tell us something about you that we don’t know’. 

We know you’re a huge football fan, but what is something about you that our audience does not know?

Outside of work I do love football. I still play (badly) in 5-a-side league and I’m a Gooner (Arsenal fan) and watch them whenever I can. 

Other than that, I love music. I have a preference for rock music, or old skool hip-hop. I used to play in bands as a drummer, one of which was called Slippery Fish. I’ve always enjoyed a silly name! I am now learning guitar and improving (very slowly).

On Aussie music, Silverchair were a band I loved growing up, and I think Neon Ballroom is such an underrated album. I am also a fan of Violent Soho who love a riff.

Thank you so much for chatting with Search It Local today Dan.

This interview was conducted by Search It Local’s Alexander Porter

Looking for more inspiration from the best and brightest minds in the industry?

Check out our interview with SEO guru and marketing expert Neil Patel.

Check out our interview with former Apple Evangelist, and current author and speaker Guy Kawasaki.