Sitebulb Crawler Tests

Chrome

Shadow DOM

• Page with Shadow DOM link
• Page with text in the Shadow DOM and links
• Page has multiple Shadow DOM elements
• Page with Custom Element Shadow DOM link
• Page with text in the Custom Element Shadow DOM and links
• Page has multiple Custom Element Shadow DOM elements
• Page with Recursive Custom Element Shadow DOM link
• Page with text in a Recursive Custom Element Shadow DOM and links
• Page has multiple Recursive Custom Element Shadow DOM elements

Iframes

• Iframe with Text and Links
• Iframe with noindex and indexifembedded
• Iframe with noindex and no indexifembedded
• Iframe with Shadow DOM
• Iframe with Canonical
• Iframe Source - Noindex
• Iframe Source - Noindex with indexifembedded

Navigation

• Page load delayed by 30 seconds

Structured Data

Vehicle listing (Car)

• Valid New Car
• Valid Used Car
• Invalid Used Car - Missing mileageFromOdometer
• Invalid New Car - Missing Brand Name
• Invalid Car - All Custom Errors

HTTP Headers

LINK

• Page with a link header for canonical, prev and next

On-Page

• Meta Description Outside Of Head
• Title Outside Of Head
• Page with Iframe that has a meta description
• Page with Iframe that has a title tag
• Page with inline SVG that has a title tag
• Page with external SVG that has a title tag
• Page with object that points to a SVG that has a title tag

Links

• Broken Content Bookmarks
• Broken Bookmark to Internal URL
• Links to Page and does not pass on PageRank
• Contains link to localhost
• Contains link to 127.0.0.1
• Contains link to local file path
• Contains link with onclick attribute
• Contains link with empty href attribute
• Contains link with no href attribute
• Contains link with no text
• Contains a link with leading whitespace character in the href
• Contains a link with trailing whitespace character in the href
• Has a mix of nofollow and follow links
• Contains a link with non-descriptive text
• Contains 13 links with different non-descriptive text
• Contains no outgoing links
• Contains a linked image with no alt text
• Contains a link to a non-HTTP protocol URL
• Contains a malformed link
• Contains a malformed link - example 2
• Contains a encoded malformed link
• Contains a large malformed link
• Contains a whitespace malformed link

Rendered vs Reponse

• All Things Changes
• JavaScript Rendered Content - Full Page
• JavaScript Rendered Content - Partial Content Update
• JavaScript Rendered Content - Appended every 1 second
• JavaScript Rendered Content - Appended every 2 second
• JavaScript Rendered Content - Appended every 3 second
• JavaScript Rendered Content - Appended every 4 second
• JavaScript Rendered Content - Appended every 5 second
• JavaScript Rendered Content - Delayed By 1 second
• JavaScript Rendered Content - Delayed By 2 seconds
• JavaScript Rendered Content - Delayed By 3 seconds
• JavaScript Rendered Content - Delayed By 4 seconds
• JavaScript Rendered Content - Delayed By 5 seconds
• JavaScript Rendered Content - Delayed By 10 seconds
• JavaScript Rendered Content - Delayed By 15 seconds
• JavaScript Rendered Content - Delayed By 30 seconds
• JavaScript Rendered Content - Delayed By 45 seconds
• JavaScript Rendered Content - Delayed By 60 seconds
• Noindex Only in Response HTML
• Nofollow Only in Response HTML
• Canonical Only in Rendered HTM
• Rendered Canonical Mismatch
• Page Title Only in Rendered HTML
• Page Title updated by JavaScript
• Meta Description Only in Rendered HTML
• URLs with Meta Description Updated by JavaScript
• URLs with H1 Only in Rendered HTML
• URLs with H1 Updated by JavaScript
• 2 JavaScript Rendered Links
• 4 JavaScript Rendered Links
• 6 JavaScript Rendered Links
• 8 JavaScript Rendered Links
• JavaScript Injected Schema

Robots Directives

• Meta Description With Whitespace
• Missing Meta Description
• Empty Meta Description
• Meta Description Too Short
• Meta Description Too Long
• Multiple Meta Descriptions
• Blocked by Robots.txt (all bots)
• data-nosnippet
• X-Robots-Tag HTTP Header all directives
• X-Robots-Tag HTTP Header Googlebot all directives
• X-Robots-Tag HTTP Header Googlebot Max-Snipet
• X-Robots-Tag HTTP Header Googlebot Nosnippet
• X-Robots-Tag HTTP Header Noarchive
• Meta Noarchive
• Meta Nocache
• Meta Noindex Googlebot

Internal URLs

URLs

• Internal Query String Contains Question Mark

Session IDs

• phpsessionid Param URL
• cid Param URL
• aspsessionid Param URL
• asp.net_sessionid Param URL
• jsessionid Param URL
• session-id Param URL

Tracking Codes

• _gl tracking Param URL
• utm_id tracking Param URL
• _ga tracking Param URL
• utm_id tracking Param URL
• utm_campaign tracking Param URL

Broken URLs

• Forbidden
• Not Found
• Error
• Timeout 5 seconds
• Timeout 30 seconds
• Timeout 45 seconds
• Timeout 60 seconds
• Timeout 75 seconds
• Timeout 150 seconds

Query string contains a question mark

• Two question marks at the start
• Question mark in quotes
• Question mark twice - same as example
• Question mark at the end of string
• Question with duplicate parameters

Query string contains more than three parameters

• 4 parameters

Query string contains paginated parameters

• page
• start
• from
• limit
• index
• pageindex
• page-index
• page as second parameter
• start as second parameter
• from as second parameter
• limit as second parameter
• index as second parameter
• pageindex as second parameter
• page-index as second parameter

Query string contains repetitive parameters

• Repetitive parameters same values
• Repetitive parameters different values

Query string contains search or filter parameters

• R as keyword search
• Query as keyword search
• Result as keyword search
• Fids as keyword search
• Refine as keyword search
• Search as keyword search
• Filter as keyword search
• keyword as keyword search
• R as keyword search
• R as keyword search (as second parameter)
• Query as keyword search (as second parameter)
• Result as keyword search (as second parameter)
• Fids as keyword search (as second parameter)
• Refine as keyword search (as second parameter)
• Search as keyword search (as second parameter)
• Filter as keyword search (as second parameter)
• keyword as keyword search (as second parameter)
• R as keyword search (as second parameter)

Query string contains sort parameters

• sby as sort parameter
• sort as sort parameter
• sortby as sort parameter
• orderby as sort parameter
• rank as sort parameter
• rank-by as sort parameter
• option as sort parameter
• sby as sort parameter (as second parameter)
• sort as sort parameter (as second parameter)
• sortby as sort parameter (as second parameter)
• orderby as sort parameter (as second parameter)
• rank as sort parameter (as second parameter)
• rank-by as sort parameter (as second parameter)
• option as sort parameter (as second parameter)

Query string contains Session ID parameters

• aspsessionid as session id
• cgisessionid as session id
• cgisessid as session id
• phpsessid as session id
• phpsession-id as session id
• phpsessionid as session id
• phpbb2mysql_sid as session id
• aspsession-id as session id
• asp.net_sessionid as session id
• jsessionid as session id
• jsessid as session id
• jsession_id as session id
• j-sessionid as session id
• sid as session id
• c-id as session id
• cfid as session id
• cftoken as session id
• cid as session id
• cust-id as session id
• hid as session id
• identity as session id
• j-sid as session id
• jid as session id
• jsid as session id
• j_id as session id
• kses as session id
• ksid as session id
• lid as session id
• p-sid as session id
• pid as session id
• procid as session id
• proc_id as session id
• secid as session id
• secure-id as session id
• secureid as session id
• seskey as session id
• sessid as session id
• session-id as session id
• session-key as session id
• sessionid as session id
• sessionkey as session id
• session_id as session id
• sidentityskey as session id
• ssid as session id
• ss_id as session id
• s_id as session id
• x-id as session id
• xid as session id
• x_id as session id
• zid as session id

Query string contains tracking parameters

• _ga as tracking parameter
• _gl as tracking parameter
• utm_id as tracking parameter
• utm_source as tracking parameter
• utm_medium as tracking parameter
• utm_term as tracking parameter
• utm_content as tracking parameter
• utm_campaign as tracking parameter

URL contains a double slash

• Double Slash
• Double Slash x 2
• Triple Slash

URL contains non-ASCII characters

• øê file name
• øê.png?大学
• þorigge.png
• 中央大学.png
• øê file html
• øê.png?大学 html
• þorigge.png html
• 中央大学.png html

URL contains repetitive elements

• Double Internal
• Doouble Internal with extra folder

URL contains upper case characters

• Upper Case and lower case
• Uppercase folder

URL contains whitespace

• Double Internal

Google Analytics codes

• Contains GA4 Code in Script
• Contains UA Code in Script

URL contains more than one Google Analytics code

• Analytics
• Ga
• Gtag
• Urchin

URL contains more than one Google Tag Manager

• Google Tag Manager

Indexability

Isolated URL

• Only found via canonical
• Only found via redirect
• Only found via noindex

Canonicals

• Canonical Is Malformed
• Canonical Is Empty
• Canonical loop
• Canonical Outside of Head
• Canonical Is Relative
• Canonical Points to Internal URL
• Canonical points to external URL
• Canonical points to another canonicalized URL
• Canonical points to a URL that is Not Found 404
• Canonicalized URL is noindex, nofollow
• Canonicalized URL
• Canonical points to a disallowed URL
• Canonical points to a noindex URL
• Canonical points to a redirecting URL
• Canonical points to a URL that is Error (5XX)
• Canonical tag in HTML and HTTP header
• Mismatched canonical tag in HTML and HTTP header
• Multiple canonical tags
• Multiple, mismatched canonical tags
• Multiple, mismatched canonical tags - in head and headers
• Canonical only found in rendered DOM
• Rendered Canonical is different to HTML source
• URL contains a form with a GET method
• Canonical points to homepage"

NoIndex NoFollow

• Multiple Robots Directives - noindex in head and index in headers
• Multiple Robots Directives - index in head and noindex in headers
• Multiple nofollow directives - multiple in head
• Multiple nofollow directives - in head and headers
• Multiple noindex directives - multiple in head
• Multiple noindex directives - in head and headers
• Mismatched nofollow directives in HTML and header - nofollow in head and follow in headers
• Mismatched nofollow directives in HTML and header - follow in head and nofollow in headers
• Noindex in HTML and HTTP header
• URL only has nofollow incoming internal links
• Meta robots found outside of <head>

Disallowed Urls

• Disallowed image
• Disallowed JavaScript
• Disallowed CSS
• Internal Disallowed URLs

Base Urls

• Base URL malformed or empty - malformed
• Base URL malformed or empty - empty
• Base URL malformed or empty - empty
• Multiple base URLs
• Multiple, mismatched base URLs

Head Issues

• <head> contains a <noscript> tag, which includes an image
• <head> contains a <noscript> tag

Soft 404s

• 404 Not Found
• 404 error
• 410 Not Found
• cannot find the page
• cannot seem to find that page
• ould not find that page
• could not find the page
• could not find what you're looking for
• can't find the page
• can't seem to find that page
• couldn't find that page
• couldn't find the page
• couldn't find what you're looking for
• error 404
• error occurred
• no products found
• 0 results
• no results found
• no results found
• no search results
• page can't be found
• page cannot be displayed
• page couldn't be found
• page doesn't exist
• page no longer exists
• page not found
• page which no longer exists
• page you are looking for cannot be found
• page you are looking for does not exist
• page you are looking for doesn't exist
• page you requested can't be found
• page you requested couldn't be found
• page you requested does not exist
• page you requested was not found
• page you were looking for is not here
• page you're looking for cannot be found
• page you're looking for does not exist
• page you're looking for doesn't exist
• requested URL was not found
• resource not found
• search returned no results
• unable to locate the requested page

International

International Tests

• Canonicalized URL has incoming hreflang
• Has conflicting outgoing hreflang annotations
• Has outgoing hreflang annotations to canonicalized URLs
• Has outgoing hreflang annotations to disallowed URLs
• Has outgoing hreflang annotations to noindex URLs
• Invalid incoming hreflang annotations
• Missing reciprocal hreflang (no return-tag)
• Hreflang annotation also x-default
• Has outgoing hreflang annotations to broken URLs
• Has outgoing hreflang annotations to redirecting URLs
• Has unsupported or misconfigured hreflang
• Has conflicting incoming hreflang annotations
• Has conflicting incoming hreflang annotations
• Has outgoing hreflang annotation to multiple URLs
• Has outgoing hreflang annotation to multiple URLs
• Has hreflang annotations without HTML lang
• Invalid HTML lang attribute
• Has hreflang annotations using multiple methods
• Has multiple self-referencing hreflang annotations
• Missing canonical URL
• Missing hreflang annotations
• Noindex URL has incoming hreflang
• Has outgoing hreflang annotations using relative URLs

Mobile Friendly

• Content does not size correctly to viewport
• Font size is too small for mobile devices
• The viewport meta tag does not have a width set
• The viewport meta tag initial-scale is incorrect
• The viewport meta tag has a maximum-scale set
• The viewport meta tag has a minimum-scale set
• The viewport meta tag prevents the user from scaling
• The viewport meta tag does not have a width set
• The viewport meta tag does not have a width set
• Missing viewport meta tag in the head
• Multiple viewport meta tags were found in the head
• Has one or more image-map map tags
• Tap targets too small

Unsupported Browser Plugins

• Director
• Flash
• Java
• Shockwave
• Silverlight

Performance

• Contains large image over 100kb