searchlove london 2017 | jon myers | mobile-first preparedness: what we've learned from...

@jondmyers @DeepCrawl

Mobile-First Preparedness- what we've learned from crawling the top 1 million websites.Jon Myers – Chief Growth Officer

Source: http://glooadvertising.com.au/is-your-website-mobile-ready/

Google is our world…

2 Trillion searches per year!

15% of searches new each day!

60% of search are on Mobiles!

Really??...

Sooooooo 2015! And before!

Google is thinking about it...

Helpful to see…

1 page at a time could take a while!…

@jondmyers @DeepCrawlSource: http://shop.adamjk.com/product/1-page-at-a-time

History of my devices…

History of Mobile…

1999 2007 2009 2014 2015 2017

Separate Mobile

Pages (Mobile Site

or Dynamic

Delivery)

Responsive

Design

Deep App

Linking

Progressive

Web Apps

Mobile-

Indexing

Time to get SH*T done…

@jondmyers @DeepCrawlSource: https://www.successimpulse.com/products/less-talk-more-action?variant=44575800839

Heavy Data ahead…

@jondmyers @DeepCrawlSource: https://betanews.com/2015/09/14/8-early-warning-signs-of-problems-in-your-data-governance-plan/

Being Majestic…

Source: https://lafleur.marketing/blog/six-mad-men-marketing-tips-don-draper/

What we did, why? And only we could…

@jondmyers @DeepCrawlSource: https://markfisherfitness.com/what-could-have-been/

History of Mobile SEO Configuration…

Desktop

Mobile

Responsive

AMP pages

Mobile first

One version for desktop devices

One to rule them all - one version designed to work equally good on desktop and mobile

Dedicated light weight version designed for a fast loading

Dedicated mobile pages served on a separate URL e.g. m.domain or dynamically served on the same URL

Mobile becomes the PRIMARY version.

Types of Mobile Configuration…

1. No Mobile Configuration (desktop only)

2. Responsive

3. Dynamic

4. Desktop + Dedicated Mobile

5. Desktop + AMP

6. Responsive/Dynamic + AMP

7. Desktop + Dedicated Mobile + AMP

8. Responsive + Dedicated Mobile + AMP

9. Responsive + Dynamic + Dedicated Mobile + AMP

10. Mobile/AMP only

11. Mobile + AMP

Impact of Mobile-first…

-CRITICAL- - Desktop only (Heavily Affected)

-HIGH PRIORITY- - Desktop + Dedicated Mobile (Affected)

-HIGH PRIORITY- - Dynamic (Affected e.g. Content Issues)

-LOW PRIORITY- - Responsive (Not affected)

Impact of Mobile-first on different Mobile Configurations:

Responsive Dynamic Separate

Mobile

Would expect Google to use both

user-agents on the same URL to

validate that the same content is

returned.

Google needs to crawl with both user

agents to validate the mobile version.

Hint: Use the Vary HTTP header!

Google needs to crawl the dedicated mobile URLs with a mobile user agent to validate the pages and confirm the content matches the desktop pages.

How does Google crawl different configurations?

Responsive design is the optimal situation…

“Responsive design is pretty much the optimal situation because regardless of which device we use to crawl the site we will see the same thing.

We’ll get the same content and the structured data, the same videos, the same images and the same internal links. It’s all there.”

Reference: https://youtu.be/7Aq9bFdfMuE?t=54m53s

How does the world look?...

Source: http://abcnews.go.com/topics/news/doomsday-clock.htm

How are sites set up for mobile?...

Mobile Configuration Breakdown…

Breakdown of all sites where mobile configuration could be identified

Significant no. dynamic sites – work to ensure meta data is kept consistent across different versions of page.

Responsive design dominates -Path of least resistance moving to Mobile-first Index

Separate Mobile unpopular –high maintenance to ensure desktop/mobile equivalence

15,000 sites

190,000 sites

795,000 sites

Separate Mobile Sites – Are they up to scratch?...

DesktopPage

Mobile Page

Rel=canonical

61% of Separate Mobile sites have correct canonical to desktop page.

DesktopPage

Mobile Page

No canonical

21% of Separate Mobile sites had no canonical to desktop equivalent.

DesktopPage

Mobile Page

Other page?

18% of Separate Mobile sites had canonicals pointing to wrong page

Separate Mobile Sites – Desktop/Mobile Equivalence…

Of 15,000 Separate Mobile sites returning 200 response codes:

DesktopPage

Mobile Page

Matching title & meta descriptions

DesktopPage

Mobile Page

Matching meta descriptions

DesktopPage

Mobile Page

Matching title

Separate Mobile Issues…

1%of separate mobile sites had noindex

meta tag

25% of all Separate Mobile sites

returned a non-200 status code

21% returned 3xx redirection

2% Separate Mobile sites returned

4xx errors

1% returned 5xx errors

Google pushing for responsive web…

https://webmasters.googleblog.com/2017/09/how-to-move-from-

m-dot-urls-to.html

Separate Mobile

Google needs to crawl the dedicated mobile URLs with a mobile user agent to validate the pages and confirm the content matches the desktop pages.

Order of Mobile-first Indexing…

“If you have a vary header which returns a different page for a mobile user agent, Google will use that as the mobile page instead of the responsive page.”

Reference:https://www.youtube.com/watch?v=8NyZypIfOzI&t=36m31s

Dynamic configurations…

Initial analysis found ?% of websites that serve

different content based on user agent do not have

a Vary: User-agent header.

Dynamic – same URL, different HTML

dependent on user agent

Let’s talk about speed…

Source: https://play.google.com/store/apps/details?id=com.ea.game.nfs14_row

Fetch times – The need for speed…

Percentage of sites split by fetch time

Fetch time includes time taken to fetch URL and display the HTTP response. This doesn’t

include the time taken to request or run any associated resources (such as images or scripts) on

the page.

But are the top million sites really that fast?...

Fetch times – Configuration breakdown…

Dynamically servedResponsive Separate Mobile

Average fetch time by configuration (secs):

Separate mobile sites have the slowest fetch times.Responsive sites have fastest fetch times of

three mobile configurations.

Fetch time with mobile desktop user agents…

19% of sites had a

fetch time between 1-

2 seconds.

68% of sites had a

fast fetch time – below

one second.

13% of sites had a

slow fetch time

exceeding 2 seconds

What does Google have to say about site performance?...

Source: https://www.thinkwithgoogle.com/about/

Google research – Performance per vertical…

Source: https://www.thinkwithgoogle.com/marketing-resources/data-measurement/mobile-page-speed-new-industry-benchmarks/

Consumer sites

have faster more

responsive web

servers.

Business and

finance lagging

behind.Technology my

Google research – Performance per vertical…

Google says to aim for 2.4 seconds load time for a

page!!!???!!

Lets Talk Page Size…

Google research – Page Size…

According to Google, sites

across all verticals are on

average larger than the

500kb recommendation.

Again, technology sites

aren’t leading the way.

What did we find out about HTML size?...

94%<=200k

5%200k - 500k

Accelerated Mobile Pages…

Percentage of sites with AMP version of homepage:

Responsive0.63%

Dynamic

0.61%Separate Mobile

What about HTTPS?...

HTTPS – How many have made the switch?...

Has HTTPS reached critical mass? Our data

says there’s a long way to go for the top one

million sites.

This figure is much lower than currently reported

Mozcast – 68% of first page results have SSL

encryption.

Source: http://mozcast.com/features

HTTPS – Adoption across mobile configurations…

Dynamically served sites appear

to be particularly slow to adopt

HTTPS.

Surprisingly Separate Mobile sites

leading the way for the switch to

HTTPS.

HTTPS doesn’t end there…

Watch out for non-secure form fields (we

found a lot of them) as Chrome is now

flagging these with security warnings.

12.7% incorrectly configured sites with links

between protocols and some sites with mixed

content e.g. a HTTPS page with reference to a

script on an HTTP URL.

Carving up Configurations by Country…

Splitting the Majestic Million by TLDs…

Percentage of sites per TLD

.edu & .gov are leading the way with responsive design…

…and .co.uk TLDs aren’t far behind…

ResponsiveDynamic Separate

What next? Plenty of issues…

Non Secure Form Fields

Pages without Valid Canonical

Failed URLs

Empty Pages

Max Fetch Time

Thin Pages

Missing H1 Tags

Non-301 Redirects

Pages with Duplicate Body

Duplicate Pages

Malformed URLs

High External Linking

Max URL Length

Missing Titles

Pages with Duplicate Titles

No Descriptions & No Snippets

Unauthorised Pages

5xx Errors

Max Links

Broken Pages (4xx Errors)

Max Description Length

Short Titles

Unlinked Paginated Pages

Hreflang to Non-200 URLs

Non-rel Alted AMP Pages

All Broken Links

Non-reciprocal Mobile/AM

Duplicate Description Sets

Max Content Size

Uncategorised HTTP Response Codes

Max Title Length

Duplicate Body Sets

Duplicate Page Sets

Pages with Duplicate Descriptions

Canonical to Non-200

Max Redirections

Max HTML Size

Short Descriptions

Redirect Loops

Duplicate Title Sets

And more…

Non-200 Mobile/AMP

Excessive Redirects In (Admin Only)

Mobile Links Out Mismatch

Mobile Links In Mismatch

Mobile Word Count Mismatch

Mobile Content Mismatch

Duplicate Pages including Primary

In a nutshell…

Separate Mobile sites are poorly configured

Dynamic sites need to check they have Vary HTTP response header…

HTTPS adoption relatively low amongst top million sites…

Fetch time is surprisingly low???…

Watch this space for an even deeper dive into the data…

Source: http://dive-bohol.com/

But what does the future hold?...

Go beyond Responsive with PWA’s…

Progressive Web Apps is:

• Progressive

• Responsive

• App-like

• And more...

Progressive Web Apps can fall back to AMP. Hint… first load as AMP.

You should think about it now.

Progressively Web App is THE FUTURE.

Jon MyersDeepCrawl

Chief Growth Officer

searchlove london 2017 | jon myers | mobile-first preparedness: what we've learned from...

Marketing

rock your brand - melanie spring - searchlove 2014

searchlove boston 2015 | luke summerfield, 'brain-based...

searchlove boston 2015 | craig bradford, 'creating digital...

searchlove london 2016 | tom anthony | seo split-testing -...

searchlove boston 2013_eppie vojt_scaling outreach

5 benefits of web crawling services over in-house crawling

not your father's google news searchlove 2015

advanced crawling techniques chapter 6. outline selective...

searchlove london 2015 | talia wolf | emotional targeting

mad*pow's marli mesibov at searchlove 2015

optimizing for engagement (for searchlove boston)

searchlove boston 2013_wil reynolds_how we get unstuck

searchlove 2013 - getting unstuck

slow your roll - margot bloomstein - searchlove boston

searchlove london | hannah smith, existential crisis...

2016 searchlove frameworks for content success

searchlove boston 2013_will critchlow_technical seo

how seogadget builds links - searchlove london 2012

albert crawling

what we've seen, what we've done 2014