--- /dev/null
+# SEO Manifesto\r
+\r
+A comprehensive resource on search engines, their query parameters,\r
+and how to use them to narrow down results or achieve SEO goals.\r
+\r
+This is generally advertised as being useful for advertisers,\r
+but it is also generally useful for researchers who really need\r
+to narrow search results quickly and specifically. Even sites with search\r
+queries don't provide as advanced of query languages as many modern search\r
+engines do.\r
+\r
+It is also useful for web programmers in general who either deal\r
+with search engines and web scraping directly or need to understand\r
+the language of APIs/frameworks that do this work for them so that\r
+they can make their software much more efficient.\r
+\r
+It would also be useful for meta-search engine programmers to expose\r
+query parameters in a cross-engine way. I've, surprisingly, not seen any meta-search\r
+engine do this yet. It would be really cool if some of them already did\r
+this. If not, it would also be a cool future project!\r
+\r
+# General Resources\r
+\r
+* [SEOsly](https://seosly.com/)\r
+* [Google Developer's Search Docs](https://developers.google.com/search/docs)\r
+\r
+# Google\r
+\r
+## Endpoints\r
+\r
+Google's main endpoint for its search engine is [https://google.com/search](https://google.com/search)\r
+\r
+Google also has several advanced search pages which utilize different search providers, though they all are utility pages which are wrappers around the same ``/search`` endpoint.\r
+* [Advanced Search](https://google.com/advanced_search)\r
+* [Advanced Video Search](https://www.google.com/advanced_video_search)\r
+* [Advanced Image Search](https://www.google.com/advanced_image_search)\r
+\r
+In addition, Google has a programmable [Custom Search Engine (CSE)](https://developers.google.com/custom-search) that can be programmed from the [REST API](https://developers.google.com/custom-search/v1) as well as a [Web Dev API](https://developers.google.com/custom-search/docs/tutorial/implementingsearchbox) which is helpful to reference for clarification on more obscure query parameters.\r
+\r
+Query parameters are provided as a query string in ``GET`` requests, one of which is exemplified below.\r
+\r
+```\r
+GET https://google.com/search?q=cute+dogs&as_qdr=d\r
+```\r
+\r
+## Query parameters\r
+\r
+* \r
+* as\_q -> Alias to ``q``, Main Query\r
+* as\_epq -> Return results for exact word/phrase\r
+* as\_oq -> Return results which contain any of the words \r
+* as\_eq -> Return results which excludes all of the words\r
+* as\_nlo -> Return results with numbers having this as the low bound\r
+* as\_nhi -> Return results with numbers having this as the high bound\r
+* lr -> Return results written in this language\r
+* cr -> Return results written in this country\r
+* as\_qdr -> Return results written from this previous date range up to today\r
+ - d``N`` -> Last ``N`` days\r
+* as\_sitesearch -> Return only results from this site\r
+* as\_occt -> Search only this type of text when returning results\r
+ - url -> Search URL of resource\r
+ - link -> Search links inside resource\r
+ - text -> Search text inside resource\r
+ - title -> Search title of resource\r
+* as\_filetype -> Return results with this file extension or filetype\r
+ - Does not include ``.``\r
+ - Unknown whether the actual type of file or just its extension is considered\r
+* imgsz -> Return images with this size\r
+ - l -> Large\r
+* imgar -> Return images with this aspect ratio\r
+ - t -> Tall\r
+* imgc -> Return images matching this color scheme\r
+ - gray -> Monochrome\r
+* imgcolor -> Return images matching this color\r
+ - red -> Red\r
+* imgtype -> Return images of this type\r
+ - face -> Face pictures\r
+* tbs -> Comma-separated extra parameters\r
+ - sur:f -> Free to use and share not including commercially\r
+ - sur:fm -> Free to use, share, and modify not including commercially\r
+ - sur:fc -> Free to use and share including commercially\r
+ - sur:fmc -> Free to use, share, and modify including commercially\r
+ - sur:cl -> Creative Commons License\r
+ - sur:ol -> Commercial & Other Licenses\r
+ - dur:s -> Return only videos with a short duration (0-4 minutes)\r
+ - dur:m -> Return only videos with a medium duration (4-20 minutes)\r
+ - dur:l -> Return only videos with a long duration (20+ minutes)\r
+ - hq:h -> Return only videos in HD\r
+ - cc:1 -> Return only videos with closed captioning\r
+* tbm -> User-friendly name for service provider\r
+ - isch -> Google Images\r
+ - vid -> Google Videos\r
+ - nws -> Google News\r
+ - bks -> Google Books\r
+ - flm -> Google Flights\r
+* udm -> Not user-friendly ID for service provider\r
+ - 2 -> Google Images\r
+ - 14 -> Google Web\r
+ - 18 -> Google Forums\r
+ - 28 -> Google Shopping\r
+* q -> Main Query (Series of operations)\r
+ - ``WORD`` -> Return results containing ``WORD``\r
+ - -``OP`` -> Return only results where ``OP`` is false\r
+ - ``OP1`` ``OP2`` -> Return results matching operations ``OP1`` and ``OP2``\r
+ - ``OP1`` OR ``OP2`` -> Return results matching operations ``OP1`` or ``OP2``\r
+ - (``OP``) -> Evaluate ``OP`` first in order of operations\r
+ - "``OP``" -> Do not delimit operations in ``OP`` by spaces\r
+ - [``OP``] -> Alias for "``OP``"\r
+ - site:``SITE`` -> Only return results from ``SITE``\r
+ - before:``DATE`` -> Only return results before ``DATE``\r
+ * ``DATE`` is in the form YEAR-MONTH-DAY\r
+ - after:``DATE`` -> Only return results after ``DATE``\r
+ * ``DATE`` is in the form YEAR-MONTH-DAY\r
+ - filetype:``EXT`` -> Return results of filetype ``EXT``\r
+ * If result does not end in ``.EXT`` but is a file of type ``EXT``, I do not know if that result is still returned, and vice versa.\r
+ * Otherwise, results ending in ``.EXT`` are returned.\r
+ - intitle:``OP`` -> Search title instead of everything when matching ``OP``\r
+ - allintitle:``OP_1`` ... ``OP_N`` -> Search title instead of everything when matching ``OP_1``, ``OP_2``, ..., and ``OP_N``\r
+ - intext:``OP`` -> Search text instead of everything when matching ``OP``\r
+ - allintext:``OP_1`` ... ``OP_N`` -> Search text instead of everything when matching ``OP_1``, ``OP_2``, ..., and ``OP_N``\r
+ - related:``SITE`` -> Find other sites related to website ``SITE``\r
+ - \* -> Wildcard operator matching all phrases \r
+\r
+# Bing\r
+\r
+## Endpoints\r
+\r
+Bing's main endpoint for its search engine is [https://www.bing.com/search](https://www.bing.com/search)\r
+\r
+Bing also has a [Web Search API](https://learn.microsoft.com/en-us/bing/search-apis/bing-web-search/reference/endpoints)\r
+\r
+\r
+Query parameters are provided as a query string in a ``GET`` request exemplified below.\r
+```\r
+GET https://www.bing.com/search?q=Hell+is+a+place+on+earth&count=2\r
+```\r
+\r
+## Query parameters\r
+\r
+* count -> Number of results per page\r
+* q -> Main query (Series of operations)\r
+ - ``WORD`` -> Return results containing ``WORD``\r
+ - ``OP1`` ``OP2`` -> Return results containing ``OP1`` and ``OP2``\r
+ - ``OP1`` OR ``OP2`` -> Return results containing ``OP1`` or ``OP2``\r
+ - -``OP1`` -> Return only results where ``OP1`` is false\r
+ - (``OP``) -> Evaluate ``OP`` first in order of operations\r
+ - "``OP``" -> Do not delimit operations in ``OP`` by spaces\r
+ - site:``SITE`` -> Return results only from the domain ``SITE``\r
+ - language:``LANG`` -> Return results only written in the language ``LANG``\r
+ - ip:``IP`` -> Return results only from ipv4 address ``IP``\r
+ - related:``SITE`` -> Find other sites related to website ``SITE``\r
+ - filetype:``EXT`` -> Return results of filetype ``EXT``\r
+ * Supposedly, should be actually based on file contents rather than extension. Haven't tested this.\r
+ - ext:``EXT`` -> Return results with file extension ``EXT``\r
+\r
+# Robots Exclusion Protocol (robots.txt)\r
+\r
+The robots exclusions protocol is useful for excluding various URIs that innocent and well-programmed web crawlers\r
+should crawl. Note that web crawlers can craft any HTTP request to any URI that they want, so the protocol lays\r
+out only suggestions to web crawlers rather than enforcements.\r
+\r
+Web crawlers should read ``/robots.txt`` from the ``HTTP`` server and parse it according to the robots exclusion\r
+protocol. Including this file can optimize how bots index your page and can reduce automated visits that you intend\r
+to be used only by browsers.\r
+\r
+It's also useful to block certain bots. Again, someone can patch the bot anyways to have the same behavior\r
+but lie about its user agent, but it at least stops the original innocent bot from crawling your page, which\r
+would be a much larger ratio of traffic anyways.\r
+\r
+## Groups\r
+\r
+All ``robots.txt`` files are a series of user agent "groups".\r
+The first line of a group specifies the user agent, which can be\r
+a specific one (i.e. ``Googlebot``) or ``*``, which is a special\r
+character specifying all user agents.\r
+\r
+```\r
+# Specific user agent\r
+user-agent: Googlebot\r
+\r
+# All user agents\r
+user-agent: *\r
+```\r
+\r
+## Rules\r
+\r
+After the user agent for a group is specified, a series of "allow"\r
+or "disallow" rules are specified on their own lines.\r
+\r
+```\r
+disallow: /cgi-bin\r
+allow: /blog\r
+```\r
+\r
+The rules are terminated by specifying a line which isn't an\r
+``allow`` or ``disallow`` rule, or by ending the file.\r
+ \r
+## Additional directives\r
+\r
+Bots can use additional directives to guide their searches.\r
+For instance, ``Sitemaps:`` is a common directive which specifies\r
+a [Sitemaps](https://www.sitemaps.org/index.html) file.\r
+\r
+```\r
+Sitemaps: https://example.org/sitemap.xml\r
+```\r
+\r
+# Sitemaps\r
+\r
+Sitemaps advise web crawlers on good URIs to visit. This is different from ``robots.txt``\r
+which specifies sites that a crawler is allowed to visit but not what it should visit.\r
+\r
+This is useful because endpoints that may be easy to access from the web browser that\r
+still have useful content for bots can be hard to visit by bots when they start from\r
+the root and crawl from there.\r
+\r
+One particularly cool detail about sitemaps is that URIs can include query strings as well,\r
+so some ``GET`` requests that require a form to be filled out can be included in the sitemap\r
+so that the bot is able to visit your page without filling out the form.\r
+\r
+## Schema\r
+\r
+There are a couple of different schemas, one for plain text (``sitemap.txt``) and\r
+one for XML (``sitemap.xml``).\r
+\r
+With ``sitemap.txt``, all URIs are listed on their own line.\r
+\r
+```\r
+https://www.example.com/\r
+https://www.example.com/dogs\r
+https://www.example.com/cats?i=2&j=3\r
+```\r
+\r
+With ``sitemap.xml``, URIs are specified in a ``urlset`` root.\r
+\r
+```xml\r
+<?xml version="1.0" encoding="UTF-8"?>\r
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\r
+ <url><loc>https://www.example.com/</loc></url>\r
+ <url><loc>https://www.example.com/dogs</loc></url>\r
+ <url><loc>https://www.example.com/cats?i=2&j=3</loc></url>\r
+</urlset>\r
+```\r
+\r
+The core schema has other elements associated with ``<url>``, but [Google's Additional Notes about Sitemaps]\r
+(https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap#additional-notes-about-xml-sitemaps)\r
+states that Google ignores these elements or verifies them using other indicators.\r
+\r
+Any trailing slash on URIs before the query string are significant to some web servers,\r
+so they are also significant in the sitemaps. It is left out of the URI in the sitemap\r
+if the web server does not require a slash and put in if it is required.\r
+\r
+## Extensions\r
+\r
+Sitemaps can, interestingly enough, be extended depending on the crawler.\r
+For instance, sitemaps from [npr.org](https://www.npr.org) will sometimes\r
+be tailored to Google web crawlers and will contain the [Google Image Sitemap]\r
+(https://www.google.com/schemas/sitemap-image/1.1/) schema.\r
+\r
+```xml\r
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">\r
+ <!-- Rest of elements in here -->\r
+</urlset>\r
+```\r
+\r
+##\r
+\r
+# Google Search Central\r
+\r
+[Google Search Central](https://developers.google.com/search), formerly known\r
+as Google Webmasters, is a set of APIs and guides to directly improve the SEO\r
+of your site with google search. This sounds very niche and some of it is niche\r
+to improving your results with Google Search, but a lot of the advice is also\r
+good for generally optimizing across all search engines.\r
+\r
+I would like to go over this more in detail at a later point but I need to\r
+research and use it myself to give good advice on it. In any case, [SEOsly]\r
+(https://seosly.com/blog/how-to-audit-site-with-google-search-console/) has\r
+done a good job describing a lot of cool features you can do with GSC if\r
+you would like to look more into it.\r
+\r
+# Sources\r
+\r
+* [Google Developer's Search Docs](https://developers.google.com/search/docs)\r
+ - [Sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap)\r
+* [Sitemaps Specification](https://www.sitemaps.org/index.html)\r
+ - [Protocol](https://www.sitemaps.org/protocol.html)\r
+ - [FAQ](https://www.sitemaps.org/faq.html)\r
+* [RFC 9309 Robots Exclusion Protocol (robots.txt)](https://datatracker.ietf.org/doc/html/rfc9309)\r
+ - [robotstxt.org](http://www.robotstxt.org/about.html)\r
+* [Google Search](https://www.google.com/search)\r
+ - [Advanced Search](https://www.google.com/advanced_search)\r
+ - [Advanced Video Search](https://www.google.com/advanced_video_search)\r
+ - [Advanced Image Search](https://www.google.com/advanced_image_search)\r
+ - [Google REST API](https://developers.google.com/custom-search/v1)\r
+ - [Google Custom Search Overview](https://developers.google.com/custom-search/docs/overview)\r
+* [Google Developers Custom Programmable Search Engine](https://developers.google.com/custom-search)\r
+ - [JSON API](https://developers.google.com/custom-search/v1)\r
+ - [Web Dev API](https://developers.google.com/custom-search/docs/tutorial/implementingsearchbox)\r
+* [Ahref's Blog Post on Advanced Google Search Operators](https://ahrefs.com/blog/google-advanced-search-operators/)\r
+* [Bing Search](https://www.bing.com/search)\r
+ - [Bing Web Search API](https://learn.microsoft.com/en-us/bing/search-apis/bing-web-search/reference/endpoints)\r
+* [SEOsly](https://seosly.com/)\r
+ - [Bing Search Operators](https://seosly.com/blog/bing-search-operators/)\r
+ - [Google Search Operators](https://seosly.com/blog/google-search-operators/)\r
+* [Bing Advanced Search Operations](https://support.microsoft.com/en-us/topic/advanced-search-options-b92e25f1-0085-4271-bdf9-14aaea720930)\r
+* [Bing Advanced Seach Keywords](https://support.microsoft.com/en-us/topic/advanced-search-keywords-ea595928-5d63-4a0b-9c6b-0b769865e78a)\r
+* [Stack Overflow on Bing Query Parameters](https://webapps.stackexchange.com/questions/111235/what-query-parameters-does-bing-have)\r
projects / gists.git / commitdiff