This document outlines the steps that are necessary in order to make your AJAX application crawlable. Once you have fully understood each of these steps, it should not take you very long to actually make your application crawlable! However, you do need to understand each of the steps involved, so we recommend reading this guide in its entirety.

Overview of Solution

Briefly, the solution works as follows: the crawler finds a pretty AJAX URL (that is, a URL containing a #! hash fragment). It then requests the content for this URL from your server in a slightly modified form. Your web server returns the content in the form of an HTML snapshot, which is then processed by the crawler. The search results will show the original URL.

Step-by-step guide

1. Indicate to the crawler that your site supports the AJAX crawling scheme

The first step to getting your AJAX site indexed is to indicate to the crawler that your site supports the AJAX crawling scheme. The way to do this is to use a special token in your hash fragments (that is, everything after the # sign in a URL): hash fragments have to begin with an exclamation mark. For example, if your AJAX app contains a URL like this:

www.example.com/ajax.html#key=value
it should now become this:

www.example.com/ajax.html#!key=value
When your site adopts the scheme, it will be considered "AJAX crawlable." This means that the crawler will see the content of your app if your site supplies HTML snapshots.

2. Set up your server to handle requests for URLs that contain _escaped_fragment_

Suppose you would like to get www.example.com/index.html#!key=value indexed. Your part of the agreement is to provide the crawler with an HTML snapshot of this URL, so that the crawler sees the content. How will your server know when to return an HTML snapshot instead of a regular page? The answer is the URL that is requested by the crawler: the crawler will modify each AJAX URL such as

www.example.com/ajax.html#!key=value
to temporarily become

www.example.com/ajax.html?_escaped_fragment_=key=value
You may wonder why this is necessary. There are two very important reasons:

Hash fragments are never (by specification) sent to the server as part of an HTTP request. In other words, the crawler needs some way to let your server know that it wants the content for the URL www.example.com/ajax.html#!key=value (as opposed to simply www.example.com/ajax.html).
Your server, on the other hand, needs to know that it has to return an HTML snapshot, rather than the normal page sent to the browser. Remember: an HTML snapshot is all the content that appears on the page after the JavaScript has been executed. Your server's end of the agreement is to return the HTML snapshot for www.example.com/index.html#!key=value (that is, the original URL!) to the crawler.
Note: The crawler escapes certain characters in the fragment during the transformation. To retrieve the original fragment, make sure to unescape all %XX characters in the fragment. More specifically, %26 should become &, %20 should become a space, %23 should become #, and %25 should become %, and so on.

Now that you have your original URL back and you know what content the crawler is requesting, you need to produce an HTML snapshot. How do you do that? There are various ways; here are some of them:

If a lot of your content is produced with JavaScript, you may want to use a headless browser such as HtmlUnit to obtain the HTML snapshot. Alternatively, you can use a different tool such as crawljax or watij.com.
If much of your content is produced with a server-side technology such as PHP or ASP.NET, you can use your existing code and only replace the JavaScript portions of your web page with static or server-side created HTML.
You can create a static version of your pages offline, as is the current practice. For example, many applications draw content from a database that is then rendered by the browser. Instead, you may create a separate HTML page for each AJAX URL.
It's highly recommended that you try out your HTML snapshot mechanism. It's important to make sure that the headless browser indeed renders the content of your application's state correctly. Surely you'll want to know what the crawler will see, right? To do this, you can write a small test application and see the output, or you can use a tool such as Fetch as Googlebot.

To summarize, make sure the following happens on your server:

A request URL of the form www.example.com/ajax.html?_escaped_fragment_=key=value is mapped back to its original form: www.example.com/ajax.html#!key=value.
The token is URL unescaped. The easiest way to do this is to use standard URL decoding. For example, in Java you would do this:
mydecodedfragment = URLDecoder.decode(myencodedfragment, "UTF-8");
An HTML snapshot is returned, ideally along with a prominent link at the top of the page, letting end users know that they have reached the _escaped_fragment_ URL in error. (Remember that _escaped_fragment_ URLs are meant to be used only by crawlers.) For all requests that do not have an _escaped_fragment_, the server will return content as before.

3. Handle pages without hash fragments

Some of your pages may not have hash fragments. For example, you might want your home page to be www.example.com, rather than www.example.com#!home. For this reason, we have a special provision for pages without hash fragments.

Note:Make sure you use this option only for pages that contain dynamic, Ajax-created content. For pages that have only static content, it would not give extra information to the crawler, but it would put extra load on your and Google's servers.

In order to make pages without hash fragments crawlable, you include a special meta tag in the head of the HTML of your page. The meta tag takes the following form:
<meta name="fragment" content="!">
This indicates to the crawler that it should crawl the ugly version of this URL. As per the above agreement, the crawler will temporarily map the pretty URL to the corresponding ugly URL. In other words, if you place <meta name="fragment" content="!"> into the page www.example.com, the crawler will temporarily map this URL to www.example.com?_escaped_fragment_= and will request this from your server. Your server should then return the HTML snapshot corresponding to www.example.com. Please note that one important restriction applies to this meta tag: the only valid content is "!". In other words, the meta tag will always take the exact form: <meta name="fragment" content="!">, which indicates an empty hash fragment, but a page with AJAX content.

4. Consider updating your Sitemap to list the new AJAX URLs

Crawlers use Sitemaps to complement their discovery crawl. Your Sitemap should include the version of your URLs that you'd prefer to have displayed in search results, so in most cases it would be http://example.com/ajax.html#!key=value. Do not include links such as http://example.com/ajax.html?_escaped_fragment_=key=value in the Sitemap. Googlebot does not follow links that contain _escaped_fragment_! If you have an entry page to your site, such as your homepage, that you would like displayed in search results without the #!, then add this URL to the Sitemap as is. For instance, if you want this version displayed in search results:

http://example.com/
then include

http://example.com/
in your Sitemap and make sure that <meta name="fragment" content="!"> is included in the head of the HTML document. For more information, check out our additional articles on Sitemaps.

5. Optionally, but importantly, test the crawlability of your app: see what the crawler sees with "Fetch as Googlebot".

Google provides a tool that will allow you to get an idea of what the crawler sees, Fetch as Googlebot. You should use this tool to see whether your implementation is correct and whether the bot can now see all the content you want a user to see. It is also important to use this tool to ensure that your site is not cloaking.

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 3.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies.

Last updated June 18, 2014.

参考:
https://developers.google.com/webmasters/ajax-crawling/docs/getting-started
https://developers.google.com/webmasters/ajax-crawling/docs/learn-more

如何使你的Ajax应用内容可让搜索引擎爬行的更多相关文章

  1. php面试专题---13、AJAX基础内容考点

    php面试专题---13.AJAX基础内容考点 一.总结 一句话总结: ajax对提升用户速度,缓解服务器压力方面也是很有可取之处的,毕竟传递的数据少了 1.AJAX基础概念? Asynchronou ...

  2. PHPcurl抓取AJAX异步内容(转载)

    PHPcurl抓取AJAX异步内容 其实抓ajax异步内容的页面和抓普通的页面区别不大.ajax只不过是做了一次异步的http请求,只要使用firebug类似的工具,找到请求的后端服务url和传值的参 ...

  3. PHP curl 抓取AJAX异步内容

    其实抓ajax异步内容的页面和抓普通的页面区别不大.ajax只不过是做了一次异步的http请求,只要使用firebug类似的工具,找到请求的后端服务url和传值的参数,然后对该url传递参数进行抓取即 ...

  4. PHP面试 AJAX基础内容

    AJAX基础内容 Ajax的基本工作原理 Ajax基础概念:通过在后台与服务器进行少量数据交换,Ajax可以使用网页实现异步更新 Ajax工作原理:XMLHttpRequest是Ajax的基础     ...

  5. 在asp.net中使JQuery的Ajax用总结

    自从有了JQuery,Ajax的使用变的越来越方便了,但是使用中还是会或多或少的出现一些让人短时间内痛苦的问题.本文暂时总结一些在使用JQuery Ajax中应该注意的问题,如有不恰当或者不完善的地方 ...

  6. 几种方法实现ajax请求内容时使用浏览器后退和前进功能

    ajax是一个非常好玩的小东西,不过用起来也会存在一些问题. 我们可以利用ajax进行无刷新改变文档内容,但是没办法去修改URL,即无法实现浏览器的前进与后退.书签的收藏功能. 利用location的 ...

  7. Windows Phone中扩展WebBrowser使其支持绑定html内容

    在WP开发中,有时候会用到WebBrowser控件来展示一些html内容,这个控件有很多局限性,比如不支持绑定内容,这样的MVVM模式中就无法进行内容的绑定.为了实现这个目的,需要扩展一下,具体代码如 ...

  8. espcms列表页ajax获取内容 - 并初始化swiper

    <link rel="stylesheet" href="swiper.min.css" type="text/css" media= ...

  9. 诅咒JavaScript之:Jquery ajax提交内容异常

    jquery ajax 通过url提交内容,在服务器端获取却出现很奇怪的值,代码如下: ajaxurl = "aspx/logTable.ashx?action=load&Every ...

随机推荐

  1. iOS中 UIWebView加载网络数据 技术分享

    直奔核心: #import "TechnologyDetailViewController.h" #define kScreenWidth [UIScreen mainScreen ...

  2. MongoDB分组

    MongoDB三种分组方式 group(先筛选再分组,不支持分片,对数据量有所限制,效率不高) [简单分组实测150W 12.5s] mapreduce(基于js引擎,单线程执行,效率较低,适合用做后 ...

  3. Java中常用的正则表达式

    常用的正则表达式 匹配中文字符的正则表达式: [\u4e00-\u9fa5] 匹配双字节字符(包括汉字在内):[^\x00-\xff] 匹配空行的正则表达式:\n[\s| ]*\r 匹配HTML标记的 ...

  4. 清除行内元素之间的HTML空白

    原文连接:Remove Whitespace Between Inline-Block Elements 原文日期: 2013年8月27日 翻译日期: 2013年8月28日 至今我还记得年轻是在IE6 ...

  5. Hibernate与Spring的事务管理

    什么是事务 这个问题比较大,按照我的理解就是,一个事务内的n个操作,要么全部完成,一旦有一个操作有问题,那么所有的操作都全部回滚. Jdbc的事务 首先,大家已经知道了,事务说白了就是一个词----统 ...

  6. JAVA之旅(十二)——Thread,run和start的特点,线程运行状态,获取线程对象和名称,多线程实例演示,使用Runnable接口

    JAVA之旅(十二)--Thread,run和start的特点,线程运行状态,获取线程对象和名称,多线程实例演示,使用Runnable接口 开始挑战一些难度了,线程和I/O方面的操作了,继续坚持 一. ...

  7. [转]C# 之DLL调用(托管与非托管)

    每种编程语言调用DLL的方法都不尽相同,在此只对用C#调用DLL的方法进行介绍.首先,您需要了解什么是托管,什么是非托管.一般可以认为:非托管代码主要是基于win 32平台开发的DLL,activeX ...

  8. ubuntu中taglist和ctags安装使用

    1.使用命令安装ctags: 2.安装taglist 下载: http://vim.sourceforge.net/scripts/download_script.php?src_id=6416 拷贝 ...

  9. 【测试】Gunicorn , uWSGI同步异步测试以及应用场景总结

    最近使用uwsgi出了一些问题,于是测试下Gunicorn测试对比下 环境 一颗cpu 1g内存 Centos系统 Django作为后端应用,Gunicorn默认模式和异步模式,响应基本是无阻塞类型 ...

  10. com.android.ddmlib.SyncException: Read-only file system

    通过eclipse运行Android 程序到测试机时候 控制台出现如下错误: [2014-02-13 15:06:03 - MPlay] Failed to install MPlay.apk on ...