Crawler is showing only the homepage?

Published 10 months ago • 1 min read

The thing that I'm going to share with you in this email does not happen every day.

But when it does, it's frustrating.

That's why I know you'll appreciate this quick tip and will remember it when you face this issue yourself.

You're trying to crawl a website. The crawl is completed super fast. And there's only the homepage in the crawl results.

🤔🤔🤔 What did go wrong here?

(by the way, reply to this email and let me know if you've faced this issue before).

Sometimes, a crawling tool might have difficulty crawling a website. It would then show only the main page and stop.

There are 2 main reasons for that:

1️⃣ Crawling of the website is disabled in its robots.txt, and your crawling tool respects it.

How to solve it:

Change the tool configuration to ignore robots.txt directives. This will allow the crawling tool to pass through the robots.txt.

Pro tip:

Check if the crawling is disabled intentionally. If not, you have an easy SEO win right here 🤩

2️⃣ Another reason why your crawler might not work is because the website you’re trying to analyze is JavaScript-based.

How to solve it:

Configure your crawler to execute JavaScript so that it can “see” website content.

Pro tip:

with a JavaScript website, make sure that Google can crawl and index the content as well.

By the way...

In my Tech SEO Pro course, there's a full module dedicated to JavaScript SEO.

There. you'll learn the 3-step framework of JavaScript debugging (so you’ll know exactly how to do it in different settings and with any frameworks).

Check out the modules and videos included in the course (JS is module 8) and watch free previews.

💡 17 ideas for adding relevant internal links at scale

You’d never guess but: Internal links are important. Oh My God Wow GIF (And are still highly undervalued). Role of internal links ✅ Determine the importance of pages within the site ✅ Influence the ranking potential of the website pages ✅ Help Google understand the website architecture and crawl your website ✅ Internal links provide the context of your content (through anchor text) ✅ Help users navigate from one page to another Here are 17 ideas for adding internal links at scale, i.e. using...

9 days ago • 1 min read

Wow! Just amazing ...

The most valuable lesson I learnt is that I can be a genius, but if I can't communicate in ways where people can understand and digest, I'm no different than a sitting fool. Zafran Zaba, Lead SEO Strategist This is part of the Tech SEO Pro course testimonial I received the other day. And I felt inspired. It reminded me why I created the course and have been sharing so much information through my free newsletters and LinkedIn posts: 😎 Solid expertise + 🤝 proper communication = 🏆 win-win Zafran...

23 days ago • 1 min read

🤖 Don't let AI kill your technical SEO expertise

This is not your typical "AI tools and secrets for tech SEOs" post. Because the tools and secrets are secondary. The first thing is your expertise. I know, it can be an unpopular opinion. But hear me out. Every day, there's a new post on LinkedIn about "that great new AI tool for technical SEOs". It gets overwhelming as you already have a list of AI tools to try, and you never have time to get to it. So you add that another tool to your list, hoping one day you'll have time to explore it....