seo ajax crawlability in a responsive publisher world
DESCRIPTION
With the emergence of heavy javascript / AJAX heavy frameworks and the growing popularity of things like AngularJS, Ember, Backbone.js, CanJS, and even JQuery; making sites and single page apps crawlable to search engines are becoming increasingly difficult. It doesn't have to be. This presentation takes a look at some of the largest and trending publishers and some of the AJAX features they employ.TRANSCRIPT
![Page 1: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/1.jpg)
AJAX in a responsive publishing world CRAWLABILITY
![Page 2: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/2.jpg)
google.me/+EricWu
@eywu
linkedin.com/in/ericywu
github.com/eywu
![Page 3: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/3.jpg)
![Page 4: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/4.jpg)
AJAX is like violence if it doesn't solve
your problems, you’re not using
enough of it.
![Page 5: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/5.jpg)
16ms
10kb
100ms SPEED,
PERFORMANCE,
HUMAN PERCEPTION
![Page 6: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/6.jpg)
![Page 7: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/7.jpg)
PAGES CRAWLED PER DAY
80% INCREASE
20,000
80,000
May 2013 – Aug 2013
![Page 8: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/8.jpg)
80% INCREASE
ORGANIC SEARCH SESSIONS
600,000
1,200,000
May 2013 – Aug 2013
![Page 9: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/9.jpg)
Mar 2004: “Googlebot/Test” External JS
Mar 2006: Googlebot Uses Onsite Live Chat
June 2010: Caffeine (Full Rollout)
Nov 2010: Instant Preview
May 2014: GWT Fetch & Render
May 2012: Matt PSA. Don’t Block JS & CSS
Oct 2009: AJAX Crawlability _escaped_fragment_
Nov 2007: Spider’s View on Web 2.0
May 2013: Matt Video. Googlebot & AJAX
![Page 10: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/10.jpg)
SCROLL
![Page 11: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/11.jpg)
http://googlewebmastercentral.blogspot.com/2014/02/infinite-scroll-search-friendly.html
THINK BASIC
PAGINATION
NAVIGATE / PAGINATE
![Page 12: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/12.jpg)
NAVIGATE / PAGINATE
<a href="/page/2/">Load More</a>
LOAD MORE LINK
![Page 13: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/13.jpg)
NAVIGATE / PAGINATE
<a href="/page/2/">Load More</a>
LOAD MORE LINK
![Page 14: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/14.jpg)
NAVIGATE / PAGINATE
REL=NEXT / PREV
<link href="/page/10/" rel="prev" /> <link href="/page/11/" rel="canonical" /> <link href="/page/12/" rel="next" />
![Page 15: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/15.jpg)
POINTER URLS*
example.com/?last-post=42
example.com/?time=1401424133
![Page 16: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/16.jpg)
CONTINUOUS CONTENT
![Page 17: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/17.jpg)
PUSHSTATE OR SHEBANG #! http://caniuse.com/#search=history
history.pushState( {}, titleOfPage, newURL
); history.replaceState();
![Page 18: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/18.jpg)
AJAX GALLERIES
![Page 19: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/19.jpg)
CONTINUOUS CONTENT
1. PUSHSTATE
2. VIEW-SOURCE
NEXT SLIDE LINKS
3. REL=NEXT/ PREV
![Page 20: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/20.jpg)
DEFERED IMAGE LOADING
I WISH CRAWLERS SUPPORTED
REL=CANONICAL HTTP HEADERS FOR IMAGES
![Page 21: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/21.jpg)
80% DECREASE
IMAGE SEARCH SESSIONS
400,000
800,000
Nov 2012 – Mar 2013
![Page 22: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/22.jpg)
LAZY LOADING • 1x1 “BLANKS”
• SKELETON SCREENS
• LOW RESOLUTION Z Z
Z
![Page 23: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/23.jpg)
RESPONSIVE IMAGES • <PICTURE>
• SRCSET
• POLYFILL
• UA DETECTION
![Page 24: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/24.jpg)
NOSCRIPT
<img data-src='<480:small.jpg, <960:medium.jpg, >960:large.jpg' />
<noscript> <img src='full.jpg' /> </noscript>
![Page 25: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/25.jpg)
CRAWLABLE PARTIALS
![Page 26: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/26.jpg)
![Page 27: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/27.jpg)
CRAWLABLE PARTIALS LYRIC SEARCH LINKS TO http://rock.rapgenius.com/1884632/Pharrell-williams-happy/ It-might-seem-crazy-what-im-bout-to-say-sunshine-shes-here-you-can-take-a-break
![Page 28: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/28.jpg)
CRAWLABLE PARTIALS JS ENABLED REDIRECTS http://rock.rapgenius.com/Pharrell-williams-happy-lyrics#note-1884632
![Page 29: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/29.jpg)
CRAWLABLE PARTIALS JS DISABLED RENDERS http://rock.rapgenius.com/1884632/Pharrell-williams-happy/It-might-seem-crazy-what-im-bout-to-say-sunshine-shes-here-you-can-take-a-break
![Page 30: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/30.jpg)
4X PAGES CRAWLED PER DAY
QUADRUPALED
75,000
300,000
May 2013 – Aug 2013
![Page 31: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/31.jpg)
+1 MILLION
ORGANIC SEARCH SESSIONS
2,000,000
4,000,000
May 2013 – Aug 2013
![Page 32: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/32.jpg)
Technical SEO Community
HelpDeskHangouts.com
SearchSignals.com
bit.ly/ajax-crawlability
![Page 33: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/33.jpg)
APPENDIX
![Page 34: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/34.jpg)
ARCHITECTURE PRERENDER SERVERSIDE VS
![Page 35: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/35.jpg)
PRERENDER
![Page 36: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/36.jpg)
PRERENDER
![Page 37: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/37.jpg)
PRERENDER
SEO.js
CONS PROS • SINGLE VIEW TEMPLATE
• SINGLE ROUTING
• REVERSE PROXY CACHING
• _ESCAPED_FRAGMENT_
• INTERPOLATED SOURCE
![Page 38: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/38.jpg)
SERVER SIDE
![Page 39: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/39.jpg)
SERVER SIDE
![Page 40: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/40.jpg)
SERVER SIDE
![Page 41: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/41.jpg)
SERVER SIDE
![Page 42: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/42.jpg)
SERVER SIDE
![Page 43: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/43.jpg)
SERVER SIDE
![Page 44: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/44.jpg)
SERVER SIDE RENDER
CONS PROS • CLEAN URLS*
• ALL REQUESTS ARE EQUAL
• TRUE DYNAMIC RESPONSE
• 2 ROUTERS
• 2 VIEW TEMPLATES*
![Page 45: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/45.jpg)
Quote conflated from my favorite ruby XML parser » http://nokogiri.org/ Speed, Performance, and Human Perception » https://www.youtube.com/watch?v=7ubJzEi3HuA SERoundtable Timeline Links » http://www.seroundtable.com/google-javascript-webmaster-tools-18602.html Googlebot/Test External JS » http://www.seroundtable.com/archives/000236.html Googlebot Uses Onsite Live Chat » http://www.seroundtable.com/archives/003492.html Spider’s View on Web 2.0 » http://googlewebmastercentral.blogspot.com/2007/11/spiders-view-of-web-20.html AJAX Crawlability Proposal » http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html Caffine Rollout » http://googlewebmastercentral.blogspot.com/2010/06/our-new-search-index-caffeine.html Instant Previews »
http://googleblog.blogspot.com/2010/11/beyond-instant-results-instant-previews.html http://googlewebmastercentral.blogspot.com/2010/11/instant-previews.html
http://googlewebmastercentral.blogspot.com/2011/11/get-post-and-safely-surfacing-more-of.html https://sites.google.com/site/webmasterhelpforum/en/faq-instant-previews Matt Cutts PSA: Don’t Block JS & CSS » http://www.seroundtable.com/googlebot-javascript-css-14930.html Matt Video: How Does Googlebot handle content loaded via AJAX? » https://www.youtube.com/watch?v=_6mtiwQ3nvw
REFERENCES
![Page 46: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/46.jpg)
GWT Fetch & Render » http://googlewebmastercentral.blogspot.com/2014/05/rendering-pages-with-fetch-as-google.html Google Blog: Infinite Scroll Recommendations & Example » http://googlewebmastercentral.blogspot.com/2014/02/infinite-scroll-search-friendly.html LA Times Reimagined by Code and Theory » http://www.codeandtheory.com/things-we-make/the-los-angeles-times-reimagined Google Blog: Specify your canonical » http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html Google Blog: Pagination with rel=“next” and rel=“prev” » http://googlewebmastercentral.blogspot.com/2011/09/pagination-with-relnext-and-relprev.html Google Blog: Video about Pagination » http://googlewebmastercentral.blogspot.com/2012/03/video-about-pagination-with-relnext-and.html One Page Wonder: Coverage on QZ » http://www.foliomag.com/2013/one-page-wonder-infinite-scroll The Next Web Redesign Coverage » http://www.niemanlab.org/2012/10/the-next-web-redesigns-to-be-more-app-like/ The Next Web Press Release » http://thenextweb.pr.co/010a893a11df2bb61d981b2b0607c1b6784a5ab07b5ab100790b2bb3168a35f8
REFERENCES
![Page 47: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/47.jpg)
USA Today Redesign » http://blog.f-i.com/usatoday-com-redesigning-one-of-americas-most-popular-news-site/ http://designenvy.aiga.org/usa-today-website-redesign-fantasy-interactive/ http://www.businessinsider.com/usa-todays-homepage-redesigns-2012-9
Gawker 1 Year Later Success » http://thenextweb.com/insider/2012/02/02/remember-that-gawker-redesign-a-years-worth-of-data-says-it-worked/ http://www.businessinsider.com/nick-denton-loses-bet-that-the-gawker-redesign-wouldnt-hurt-traffic-2011-10 http://www.businessinsider.com/gawker-media-traffic-numbers-2011-4 Gawker Failed Coverage » http://www.catchmyfame.com/2013/05/02/how-gawker-sabotaged-their-own-network-with-a-horrible-new-layout/ http://www.theatlantic.com/technology/archive/2011/04/gawkers-traffic-numbers-are-worse-than-anyone-anticipated/237594/ http://www.webmonkey.com/2011/02/gawker-learns-the-hard-way-why-hash-bang-urls-are-evil/ Paul Irish to Matt Cutts Video » https://www.youtube.com/watch?v=yiAF9VdvRPw Google Developer Documentation on AJAX Crawlability » https://developers.google.com/webmasters/ajax-crawling/ Browser Compatibility Chart » http://caniuse.com/#search=history Breaking The Web With Hash Bangs » http://isolani.co.uk/blog/javascript/BreakingTheWebWithHashBangs
REFERENCES
![Page 48: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/48.jpg)
Vox Cards: Legalization of Marijuana » http://www.vox.com/cards/marijuana-legalization/learn-more-about-marijuana-legalization Bing’s Duane Forrester says still no rel=canonical in http headers » https://twitter.com/DuaneForrester/status/459387860358295552 Google Blog: A Faster Image Search » http://googlewebmastercentral.blogspot.com/2013/01/faster-image-search.html Google Says It’s Better for Webmasters » http://www.seroundtable.com/google-image-search-design-16259.html Ilya Grigork discussion around <plaintext> injection » https://plus.google.com/+IlyaGrigorik/posts/S6j45VxNESB Vox Workflow for Creating SVG Images » http://product.voxmedia.com/2013/11/25/5426880/polygon-feature-design-svg-animations-for-fun-and-profit One Solution to Responsive Images » http://www.smashingmagazine.com/2014/02/03/one-solution-to-responsive-images/ Truly Responsive Images » http://davidwalsh.name/responsive-design AngularJS NYC Meetup: Server-side Template Rendering by HBO » http://youtu.be/iB7hfvqyZpg?t=58m20s
REFERENCES
![Page 49: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/49.jpg)
Vox Cards: Legalization of Marijuana » http://www.vox.com/cards/marijuana-legalization/learn-more-about-marijuana-legalization Bing’s Duane Forrester says still no rel=canonical in http headers » https://twitter.com/DuaneForrester/status/459387860358295552 Google Blog: A Faster Image Search » http://googlewebmastercentral.blogspot.com/2013/01/faster-image-search.html Google Says It’s Better for Webmasters » http://www.seroundtable.com/google-image-search-design-16259.html Ilya Grigork discussion around <plaintext> injection » https://plus.google.com/+IlyaGrigorik/posts/S6j45VxNESB Vox Workflow for Creating SVG Images » http://product.voxmedia.com/2013/11/25/5426880/polygon-feature-design-svg-animations-for-fun-and-profit One Solution to Responsive Images » http://www.smashingmagazine.com/2014/02/03/one-solution-to-responsive-images/ Truly Responsive Images » http://davidwalsh.name/responsive-design Serious Angular SEO » http://www.ng-newsletter.com/posts/serious-angular-seo.html AngularJS NYC Meetup: Server-side Template Rendering by HBO » http://youtu.be/iB7hfvqyZpg?t=58m20s
REFERENCES
![Page 50: SEO AJAX Crawlability in a Responsive Publisher World](https://reader034.vdocuments.us/reader034/viewer/2022052505/5566d155d8b42a1a628b4cdc/html5/thumbnails/50.jpg)
Josh Kadis Quartz on VIP Wordpress Video » http://vip.wordpress.com/2013/09/26/josh-kadis-qz-wordpress/ https://docs.google.com/file/d/0B2Z4K6ynFLg5TVdvWVV1aTRmYUU/edit?pli=1
AirBNB: Our First Node.js App » http://nerds.airbnb.com/weve-launched-our-first-nodejs-app-to-product/ AirBNB: Rendr (Backbone in the Browser and Node) » http://nerds.airbnb.com/weve-open-sourced-rendr-run-your-backbonejs-a/ StackOverflow: PushState, Backbone, and Node » http://stackoverflow.com/questions/7098130/reusing-backbone-views-routes-on-the-server-when-using-backbone-js-pushstate-for Google: How do I create an HTML Snapshot (HIJAX) » https://developers.google.com/webmasters/ajax-crawling/docs/html-snapshot
REFERENCES