Gigablast - api




            WEB SEARCH    

            DIRECTORY    

            ADVANCED    

            ADD URL    

            SYNTAX    

            ABOUT    

            BLOG    

            FAQ    

            API    
 

ADMIN    





NOTE: All APIs support both GET and POST method. If the size of your request is more than 2K you should use POST.
NOTE: All APIs support both http and https protocols.
API by pages
/search - search results page   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3qSTRINGqueryThe query to perform. See help. See the query operators below for more info. REQUIRED
4cSTRINGcollectionSearch this collection. Use multiple collection names separated by a whitespace to search multiple collections at once. REQUIRED
5nINT32number of results per query10The number of results returned. If you want more than 1000 results you must use &stream=1 so Gigablast does not run out of memory.
6sINT32first result num0Start displaying at search result #X. Starts at 0. If you want more than 1000 results in total, you must use &stream=1 so Gigablast does not run out of memory.
7showerrorsBOOL (0 or 1)show errors0Show errors from generating search result summaries rather than just hide the docid. Useful for debugging.
8showanomaliesBOOL (0 or 1)show anomalies0Show search results that only contain the query terms in some anomalous link texts.
9scBOOL (0 or 1)site cluster0Should search results be site clustered? This limits each site to appearing at most twice in the search results. Sites are subdomains for the most part, like abc.xyz.com.
10hacrBOOL (0 or 1)hide all clustered results0Only display at most one result per site.
11drBOOL (0 or 1)dedup results0Should duplicate search results be removed? This is based on a content hash of the entire document. So documents must be exactly the same for the most part.
12pssINT32percent similar dedup summaryIf document summary (and title) are this percent similar to a document summary above it, then remove it from the search results. 100 means only to remove if exactly the same. 0 means no summary deduping. You must also supply dr=1 for this to work.
13dduBOOL (0 or 1)dedup URLs0Should we dedup URLs with case insensitivity? This is mainly to correct duplicate wiki pages.
14qeBOOL (0 or 1)do query expansionIf enabled, query expansion will expand your query to include the various forms and synonyms of the query terms.
15spellBOOL (0 or 1)do spell checkingIf enabled, when Gigablast finds a spelling recommendation it will be included in the XML <spellingSuggestion> tag or JSON "spellingSuggestion" field. Default is 0 if using an XML or JSON feed, 1 otherwise.
16autospellBOOL (0 or 1)auto correct spellingIf enabled, when Gigablast is CONFIDENT of a spelling recommendation it will automatically re-perform the query with the recommended spelling. Spell checking must be enabled for this to work. This is a default value and can be overriden directly with the autospell parm in each individual http request.
17relqueriesBOOL (0 or 1)show related queries0Offer related queries at bottom of search results.
18streamCHARstream search results0Stream search results back on socket as they arrive. Useful when thousands/millions of search results are requested. Required when doing such things otherwise Gigablast could run out of memory. Only supported for JSON and XML formats, not HTML. You must use this if you want more than 1000 results.
19secsbackINT32seconds back0Limit results to pages spidered this many seconds ago. Use 0 to disable.
20sortbyCHARsort by0Use 0 to sort results by relevance, 1 to sort by most recent spider date down, and 2 to sort by oldest spidered results first.
21filetypeSTRINGfiletypeRestrict results to this filetype. Supported filetypes are pdf, doc, html xml, json, xls.
22scoresBOOL (0 or 1)get scoring info0Get scoring information for each result so you can see how each result is scored. You must explicitly request this using &scores=1 for the XML feed because it is not included by default.
23fastBOOL (0 or 1)fast results0Sacrifice some quality and result filtering for the sake of speed.
24uipSTRINGuser ipThe ip address of the searcher. We can pass back for use in the autoban technology which bans abusive IPs.
25nfINT32max number of facets to return50Max number of facets to return
26qlangSTRINGsort language preferenceDefault language to use for ranking results. Value should be any language abbreviation, for example "en" for English. Use xx to give ranking boosts to no language in particular. See the language abbreviations at the bottom of the url filters page.
27langwFLOAT32language weightUse this to override the default language weight for this collection. The default language weight can be set in the search controls and is usually something like 20.0. Which means that we multiply a result's score by 20 if from the same language as the query or the language is unknown.
28tmlINT32max title lenWhat is the maximum number of characters allowed in titles displayed in the search results?
29nsINT32number of summary excerptsHow many summary excerpts to display per search result?
30swINT32max summary line width<br> tags are inserted to keep the number of chars in the summary per line at or below this width. Also affects title. Strings without spaces that exceed this width are not split. Has no affect on xml or json feed, only works on html.
31smxcplINT32max summary excerpt lengthWhat is the maximum number of characters allowed per summary excerpt?
32dsrtINT32results to scan for gigabits generationHow many search results should we scan for gigabit (related topics) generation. Set this to zero to disable gigabits!
33iprBOOL (0 or 1)ip restriction for gigabitsShould Gigablast only get one document per IP domain and per domain for gigabits (related topics) generation?
34nrtINT32number of gigabits to show11What is the number of gigabits (related topics) displayed per query? Set to 0 to save a little CPU time.
35mtsINT32min topics scoreGigabits (related topics) with scores below this will be excluded. Scores range from 0% to over 100%.
36mdcINT32min gigabit doc count by default2How many documents must contain the gigabit (related topic) in order for it to be displayed.
37dspINT32dedup doc percent for gigabits (related topics)80If a document is this percent similar to another document with a higher score, then it will not contribute to the gigabit generation.
38mwptINT32max words per gigabit (related topic) by default6Maximum number of words a gigabit (related topic) can have. Affects xml feeds, too.
39showimagesBOOL (0 or 1)show images1Should we return or show the thumbnail images in the search results?
40usecacheCHARuse cache1Use 0 if Gigablast should not read or write from any caches at any level.
41rcacheBOOL (0 or 1)read from cache1Should we read search results from the cache?
42wcacheCHARwrite to cache1Use 0 if Gigablast should not write to any caches at any level.
43minserpdocidINT64max serp docid0Start displaying results after this score/docid pair. Used by widget to append results to end when index is volatile.
44maxserpscoreFLOAT64max serp score0Start displaying results after this score/docid pair. Used by widget to append results to end when index is volatile.
45linkSTRINGrestrict search to pages that link to this urlThe url which the pages must link to.
46sitesSTRINGrestrict results to these sitesReturned results will have URLs from these space-separated list of sites. Can have up to 200 sites. A site can include sub folders. This is allows you to build a Custom Topic Search Engine.
47ffBOOL (0 or 1)family filter0Remove objectionable results if this is enabled.
48qhBOOL (0 or 1)highlight query terms in summaries1Use to disable or enable highlighting of the query terms in the summaries.
49hqSTRINGcached page highlight queryHighlight the terms in this query instead.
50bqINT32boolean status2Can be 0 or 1 or 2. 0 means the query is NOT boolean, 1 means the query is boolean and 2 means to auto-detect.
51dtSTRINGmeta tags to displayA space-separated string of meta tag names. Do not forget to url-encode the spaces to +'s or %%20's. Gigablast will extract the contents of these specified meta tags out of the pages listed in the search results and display that content after each summary. i.e. &dt=description will display the meta description of each search result. &dt=description:32+keywords:64 will display the meta description and meta keywords of each search result and limit the fields to 32 and 64 characters respectively. When used in an XML feed the <display name="meta_tag_name">meta_tag_content</> XML tag will be used to convey each requested meta tag's content.
52nicenessINT32niceness0Can be 0 or 1. 0 is usually a faster, high-priority query, 1 is a slower, lower-priority query.
53debugCHARdebug flag0Is 1 to log debug information, 0 otherwise.
54rdcBOOL (0 or 1)return number of docs per topic1Use 1 if you want Gigablast to return the number of documents in the search results that contained each topic (gigabit).
55rdBOOL (0 or 1)return docids per topic0Use 1 if you want Gigablast to return the list of docIds from the search results that contained each topic (gigabit).
56debuggigabitsBOOL (0 or 1)debug gigabits flag0Is 1 to log gigabits debug information, 0 otherwise.
57dioBOOL (0 or 1)return docids only0Is 1 to return only docids as query results.
58adminBOOL (0 or 1)admin override1admin override
59prependSTRINGprependprepend this to the supplied query followed by a |.
60sbBOOL (0 or 1)show banned pages0show banned pages
61iccINT32include cached copy of page0Will cause a cached copy of content to be returned instead of summary.

Example XML Output (&format=xml)

<response>
	<statusCode>0</statusCode>
	<statusMsg>Success</statusMsg>
	<currentTimeUTC>1404513734</currentTimeUTC>
	<responseTimeMS>284</responseTimeMS>
	<docsInCollection>226</docsInCollection>
	<hits>193</hits>
	<moreResultsFollow>1</moreResultsFollow>
	<result>
		<imageBase64>/9j/4AAQSkZJRgABAQAAAQABA...</imageBase64>
		<imageHeight>350</imageHeight>
		<imageWidth>223</imageWidth>
		<origImageHeight>470</origImageHeight>
		<origImageWidth>300</origImageWidth>
		<title><![CDATA[U.S....]]></title>
		<sum>Department of the Interior protects America's natural resources and</sum>
		<url><![CDATA[www.doi.gov]]></url>
		<size>  64k</size>
		<docId>34111603247</docId>
		<site>www.doi.gov</site>
		<spidered>1404512549</spidered>
		<firstIndexedDateUTC>1404512549</firstIndexedDateUTC>
		<contentHash32>2680492249</contentHash32>
		<language>English</language>
	</result>
</response>

Example JSON Output (&format=json)

{ "response":{

	# This is zero on a successful query. 
	# Otherwise it will be a non-zero number 
	# indicating the error code.
	"statusCode":0,

	# Similar to above, this is "Success" 
	# on a successful query. Otherwise it 
	# will indicate an error message 
	# corresponding to the statusCode above.
	"statusMsg":"Success",

	# This is the current time in UTC in 
	# unix timestamp format (seconds since 
	# the epoch) that the server has when 
	# generating this JSON response.
	"currentTimeUTC":1404588231,

	# This is how long it took in 
	# milliseconds to generate the JSON 
	# response from reception of the request.
	"responseTimeMS":312,

	# This is how many matches were 
	# excluded from the search results 
	# because they were considered 
	# duplicates, banned, had errors 
	# generating the summary, or were from an 
	# over-represented site. To show them use 
	# the &sc &dr &pss &sb and &showerrors 
	# input parameters described above.
	"numResultsOmitted":3,

	# This is how many shards failed to 
	# return results. Gigablast gets results 
	# from multiple shards (computers) and 
	# merges them to get the final result 
	# set. Some times a shard is down or 
	# malfunctioning so it will not 
	# contribute to the results. So If this 
	# number is non-zero then you had such a 
	# shard.
	"numShardsSkipped":0,

	# This is how many shards are ideally 
	# in use by Gigablast to generate search 
	# results.
	"totalShards":159,

	# This is how many total documents are 
	# in the collection being searched.
	"docsInCollection":226,

	# This is how many of those documents 
	# matched the query.
	"hits":193,

	# This is 1 if more search results are 
	# available, otherwise it is 0.
	"moreResultsFollow":1,

	# Start of query-based information.
	"queryInfo":{

		# The entire query that was received, 
		# represented as a single string.
		"fullQuery":"test",

		# The language of the query. This is 
		# the 'preferred' language of the search 
		# results. It is reflecting the &qlang 
		# input parameter described above. Search 
		# results in this language (or an unknown 
		# language) will receive a large boost. 
		# The boost is multiplicative. The 
		# default boost size can be overridden 
		# using the &langw input parameter 
		# described above. This language 
		# abbreviation here is usually 2 letter, 
		# but can be more, like in the case of 
		# zh-cn, for example.
		"queryLanguageAbbr":"en",

		# The language of the query. Just 
		# like above but the language is spelled 
		# out. It may be multiple words.
		"queryLanguage":"English",

		# List of space separated words in 
		# the query that were ignored for the 
		# most part. Because they were common 
		# words for the query language they are 
		# in.
		"ignoredWords":"to the",

		# There is a maximum limit placed on 
		# the number of query terms we search on 
		# to keep things fast. This can be 
		# changed in the search controls.
		"queryNumTermsTotal":52,
		"queryNumTermsUsed":20,
		"queryWasTruncated":1,

		# The start of the terms array. Each 
		# query is broken down into a list of 
		# terms. Each term is described here.
		"terms":[

			# The first query term in the JSON 
			# terms array.
			{

			# The term number, starting at 0.
			"termNum":0,

			# The term as a string.
			"termStr":"test",

			# The term frequency. An estimate of 
			# how many pages in the collection 
			# contain the term. Helps us weight terms 
			# by popularity when scoring the results.
			"termFreq":425239458,

			# A 48-bit hash of the term. Used to 
			# represent the term in the index.
			"termHash48":67259736306430,

			# A 64-bit hash of the term.
			"termHash64":9448336835959712000,

			# If the term has a field, like the 
			# term title:cat, then what is the hash 
			# of the field. In this example it would 
			# be the hash of 'title'. But for the 
			# query 'test' there is no field so it is 
			# 0.
			"prefixHash64":0

			},

			# The second query term in the JSON 
			# terms array.
			{

			"termNum":1,
			"termStr":"tested",

			# The language the term is from, in 
			# the case of query expansion on the 
			# original query term. Gigablast tries to 
			# find multiple forms of the word that 
			# have the same essential meaning. It 
			# uses the specified query language 
			# (&qlang), however, if a query term is 
			# from a different language, then that 
			# language will be implied for query 
			# expansion.
			"termLang":"en",

			# The query term that this term is a 
			# form of.
			"synonymOf":"test",

			"termFreq":73338909,
			"termHash48":66292713121321,
			"termHash64":9448336835959712000,
			"prefixHash64":0
			},

			...

		# End of the JSON terms array.
		]

	# End of the queryInfo JSON structure.
	},

	# The start of the gigabits array. 
	# Each gigabit is mined from the content 
	# of the search results. The top N 
	# results are mined, and you can control 
	# N with the &dsrt input parameter 
	# described above.
	"gigabits":[

		# The first gigabit in the array.
		{

		# The gigabit as a string in utf8.
		"term":"Membership",

		# The numeric score of the gigabit.
		"score":240,

		# The popularity ranking of the 
		# gigabit. Out of 10000 random documents, 
		# how many documents contain it?
		"minPop":480,

		# The gigabit in the context of a 
		# document.
		"instance":{

			# A sentence, if it exists, from one 
			# of the search results which also 
			# contains the gigabit and as many 
			# significant query terms as possible. In 
			# UTF-8.
			"sentence":"Get a free Tested Premium Membership here!",

			# The url that contained that 
			# sentence. Always starts with http.
			"url":"http://www.tested.com/",

			# The domain of that url.
			"domain":"tested.com"
		}

		# End of the first gigabit
		},

		...

	# End of the JSON gigabits array.
	],

	# Start of the facets array, if any.
	"facets":[

		# The first facet in the array.
		{
			# The field you are faceting over
			"field":"Company",

			# How many documents in the 
			# collection had this particular field? 
			# 64-bit integer.
			"totalDocsWithField":148553,

			# How many documents in the 
			# collection had this particular field 
			# with the same value as the value line 
			# directly below? This should always be 
			# less than or equal to the 
			# totalDocsWithField count. 64-bit 
			# integer.
			"totalDocsWithFieldAndValue":44184,

			# The value of the field in the case 
			# of this facet. Can be a string or an 
			# integer or a float, depending on the 
			# type described in the gbfacet query 
			# term. i.e. gbfacetstr, gbfacetint or 
			# gbfacetfloat.
			"value":"Widgets, Inc.",

			# Should be the same as 
			# totalDocsWithFieldAndValue, above. 
			# 64-bit integer.
			"docCount":44184

		# End of the first facet in the array.
		}

	# End of the facets array.
	],

	# Start of the JSON array of 
	# individual search results.
	"results":[

		# The first result in the array.
		{

		# The title of the result. In UTF-8.
		"title":"This is the title.",

		# A DMOZ entry. One result can have 
		# multiple DMOZ entries.
		"dmozEntry":{

			# The DMOZ category ID.
			"dmozCatId":374449,

			# The DMOZ direct category ID.
			"directCatId":1,

			# The DMOZ category as a UTF-8 
			# string.
			"dmozCatStr":"Top: Computers: Security: Malicious 
			 Software: Viruses: Detection and Removal Tools: 
			 Reviews",

			# What title some DMOZ editor gave 
			# to this url.
			"dmozTitle":"The DMOZ Title",

			# What summary some DMOZ editor gave 
			# to this url.
			"dmozSum":"A great web page.",

			# The DMOZ anchor text, if any.
			"dmozAnchor":"",

		# End DMOZ entry.
		},

		# The content type of the url. Can be 
		# html, pdf, text, xml, json, doc, xls or 
		# ps.
		"contentType":"html",

		# The summary excerpt of the result. 
		# In UTF-8.
		"sum":"Department of the Interior protects America's natural resources.",

		# The url of the result. If it starts 
		# with http:// then that is omitted. Also 
		# omits the trailing / if the urls is 
		# just a domain or subdomain on the root 
		# path.
		"url":"www.doi.gov",

		# The hopcount of the url. The 
		# minimum number of links we would have 
		# to click to get to it from a root url. 
		# If this is 0 that means the url is a 
		# root url, like http://www.root.com/.
		"hopCount":0,

		# The size of the result's content. 
		# Always in kilobytes. k stands for 
		# kilobytes. Could be a floating point 
		# number or and integer.
		"size":"  64k",

		# The exact size of the result's 
		# content in bytes.
		"sizeInBytes":64560,

		# The unique document identifier of 
		# the result. Used for getting the cached 
		# content of the url.
		"docId":34111603247,

		# The site the result comes from. 
		# Usually a subdomain, but can also 
		# include part of the URL path, like, 
		# abc.com/users/brad/. A site is a set of 
		# web pages controlled by the same 
		# entity.
		"site":"www.doi.gov",

		# The time the url was last INDEXED. 
		# If there was an error or the url's 
		# content was unchanged since last 
		# download, then this time will remain 
		# unchanged because the document is not 
		# reindexed in those cases. Time is in 
		# unix timestamp format and is in UTC.
		"spidered":1404512549,

		# The first time the url was 
		# successfully INDEXED. Time is in unix 
		# timestamp format and is in UTC.
		"firstIndexedDateUTC":1404512549,

		# A 32-bit hash of the url's content. 
		# It is used to determine if the content 
		# changes the next time we download it.
		"contentHash32":2680492249,

		# The dominant language that the 
		# url's content is in. The language name 
		# is spelled out in its entirety.
		"language":"English"

		# A convenient abbreviation of the 
		# above language. Most are two 
		# characters, but some, like zh-cn, are 
		# more.
		"langAbbr":"en"

		# If the result has an associated 
		# image then the image thumbnail is 
		# encoded in base64 format here. It is a 
		# jpg image.
		"imageBase64":"/9j/4AAQSkZJR...",

		# If the result has an associated 
		# image then what is its height and width 
		# of the above jpg thumbnail image in 
		# pixels?
		"imageHeight":223,
		"imageWidth":350,

		# If the result has an associated 
		# image then what are the dimensions of 
		# the original image in pixels?
		"origImageHeight":300,
		"origImageWidth":470

		# End of the first result.
		},

		...

	# End of the JSON results array.
	]

# End of the response.
}

}



/get - gets cached web page   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3dINT64docId0The docid of the cached page to view. REQUIRED
4urlSTRINGurlInstead of specifying a docid, you can get the cached webpage by url as well. REQUIRED
5cSTRINGcollectionGet the cached page from this collection. REQUIRED
6stripINT32strip0Is 1 or 2 two strip various tags from the cached content.
7ihBOOL (0 or 1)include header1Is 1 to include the Gigablast header at the top of the cached page, 0 to exclude the header.
8qSTRINGqueryHighlight this query in the page.

Example XML Output (&format=xml)
<response>
	<statusCode>0</statusCode>
	<statusMsg>Success</statusMsg>
	<url><![CDATA[http://www.doi.gov/]]></url>
	<docId>34111603247</docId>
	<cachedTimeUTC>1404512549</cachedTimeUTC>
	<cachedTimeStr>Jul 04, 2014 UTC</cachedTimeStr>
	<content><![CDATA[<html><title>Some web page title</title><head>My first web page</head></html>]]></content>
</response>

Example JSON Output (&format=json)
{ "response":{
	"statusCode":0,
	"statusMsg":"Success",
	"url":"http://www.doi.gov/",
	"docId":34111603247,
	"cachedTimeUTC":1404512549,
	"cachedTimeStr":"Jul 04, 2014 UTC",
	"content":"<html><title>Some web page title</title><head>My first web page</head></html>"
}
}



/admin/status - basic status   [ show parms in xml or json ]   [ show status in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionUse this collection. REQUIRED



/admin/collectionpasswords - passwords   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.



/admin/hosts - hosts status   [ show parms in xml or json ]   [ show status in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.



/admin/master - master controls   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3qrtsSTRINGquery routesdefault->127.0.0.1:9000Match a subtring in the HTTP request to the first substring before the -> below and route it, round robin style, to one host in the following comma-separated list of IP:UDP_PORT. Put an * immediately after the -> to mark the host as unpingable; just a hack for older clusters that use a different ping packet format.
4seBOOL (0 or 1)spidering enabled1Controls all spidering for all collections
5injenBOOL (0 or 1)injections enabled1Controls injecting for all collections
6qryenBOOL (0 or 1)querying enabled1Controls querying for all collections
7rraBOOL (0 or 1)return results even if a shard is down1If you turn this off then Gigablast will return an error message if a shard was down and did not return results for a query. The XML and JSON feed let's you know when a shard is down and will give you the results back any way, but if you would rather have just and error message and no results, then set then set this to 'NO'.
8maxmemINT64max mem8000000000Mem available to this process. May be exceeded due to fragmentation.
9mtspINT32max total spiders100What is the maximum number of web pages the spider is allowed to download simultaneously for ALL collections PER HOST? Caution: raising this too high could result in some Out of Memory (OOM) errors. The hard limit is currently 300. Each collection has its own limit in the spider controls that you may have to increase as well.
10aeBOOL (0 or 1)add url enabled1Can people use the add url interface to add urls to the index?
11ucpBOOL (0 or 1)use collection passwords0Should collections have individual password settings so different users can administrer different collections? If not the only the master passwords and IPs will be able to administer any collection.
12acuBOOL (0 or 1)allow cloud users0Can guest users create and administer a collection? Limit: 1 collection per IP address. This is mainly for doing demos on the gigablast.com domain.
13asfINT32auto save frequency5Save data in memory to disk after this many minutes have passed without the data having been dumped or saved to disk. Use 0 to disable.
14msINT32max http sockets300Maximum sockets available to serve incoming HTTP requests. Too many outstanding requests will increase query latency. Excess requests will simply have their sockets closed.
15mssINT32max https sockets100Maximum sockets available to serve incoming HTTPS requests. Like max http sockets, but for secure sockets.
16suaSTRINGspider user agentGigablastOpenSource/1.0Identification seen by web servers when the Gigablast spider downloads their web pages. It is polite to insert a contact email address here so webmasters that experience problems from the Gigablast spider have somewhere to vent.
17jsUNARY CMD (set to 1)saveSaves in-memory data for ALL hosts. Does Not exit. If running from a proxy, then just saves the gb.conf data for that proxy.
18saveUNARY CMD (set to 1)save & exitSaves the data and exits for ALL hosts. If running on a proxy, just saves and exits on that proxy.
19rebalanceUNARY CMD (set to 1)rebalance shardsTell all hosts to scan all records in all databases, and move records to the shard they belong to. You only need to run this if Gigablast tells you to, when you are changing hosts.conf to add or remove more nodes/hosts.
20dumpUNARY CMD (set to 1)dump to diskFlushes all records in memory to the disk on all hosts.
21pmergeUNARY CMD (set to 1)tight merge posdbMerges all outstanding posdb (index) files.
22tmergeUNARY CMD (set to 1)tight merge titledbMerges all outstanding titledb (web page cache) files.
23spmergeUNARY CMD (set to 1)tight merge spiderdbMerges all outstanding spiderdb files.
24clrkrnerrUNARY CMD (set to 1)clear kernel error messageClears the kernel error message. You must do this to stop getting email alerts for a kernel ring buffer error alert.
25afgdwdBOOL (0 or 1)ask for gzipped docs when downloading0If this is true, gb will send Accept-Encoding: gzip to web servers when doing http downloads. It does have a tendency to cause out-of-memory errors when you enable this, so until that is fixed better, it's probably a good idea to leave this disabled.
26abBOOL (0 or 1)autoban IPs which violate the queries per day quotas0Keep track of ips which do queries, disallow non-customers from hitting us too hard.
27nfqpdINT32free queries per day 1024Non-customers get this many queries per day beforebeing autobanned
28nfqpmINT32free queries per minute 30Non-customers get this many queries per minute beforebeing autobanned
29ubfBOOL (0 or 1)use bot fence0Use ajax to load the search results to keep bots from slamming gb with queries.
30bfswSTRINGsecret word for bot fencerxiwdUse ajax to set the value of this word as a cgi parm in the url. TODO: make this automatic again.
31mhdmsINT32max heartbeat delay in milliseconds0If a heartbeat is delayed this many milliseconds dump a core so we can see where the CPU was. Logs 'db: missed heartbeat by %ld ms'. Use 0 or less to disable.
32mdchINT32max delay before logging a callback or handler-1If a call to a message callback or message handler in the udp server takes more than this many milliseconds, then log it. Logs 'udp: Took %ld ms to call callback for msgType=0x%hhx niceness=%d'. Use -1 or less to disable the logging.
33errstroneSTRINGerror string 1I/O errorLook for this string in the kernel buffer for sending email alert. Useful for detecting some strange hard drive failures that really slow performance.
34errstrtwoSTRINGerror string 2Look for this string in the kernel buffer for sending email alert. Useful for detecting some strange hard drive failures that really slow performance.
35errstrthreeSTRINGerror string 3Look for this string in the kernel buffer for sending email alert. Useful for detecting some strange hard drive failures that really slow performance.
36dpcspINT64posdb disk cache size for small termlists30000000How much file cache size to use in bytes for termlists smaller than 1MB? Posdb is the index.
37balloonINT64posdb disk cache size for large termlists100000000How much file cache size to use in bytes for termlists greater than or equal to 1MB? Posdb is the index.
38dpcstINT64tagdb disk cache size30000000How much file cache size to use in bytes? Tagdb is consulted at spider time and query time to determine if a url or outlink is banned or what its siterank is, etc.
39dpcscINT64clusterdb disk cache size30000000How much file cache size to use in bytes? Gigablast does a lookup in clusterdb for each search result at query time to get its site information for site clustering. If you disable site clustering in the search controls then clusterdb will not be consulted.
40dpcsxINT64titledb disk cache size30000000How much file cache size to use in bytes? Titledb holds the cached web pages, compressed. Gigablast consults it to generate a summary for a search result, or to see if a url Gigablast is spidering is already in the index.
41dpcsyINT64spiderdb disk cache size30000000How much file cache size to use in bytes? Spiderdb holds the urls to spider along with the results of spidering a url, like the error code, bytes downloaded, etc. This cache works on spiderdb and could speed up spidering if you increase it.
42dnsmcmINT32dns cache size30000000Bytes to use for caching dns replies per host.
43wlmcmINT32winner list cache size30000000Bytes to use for caching urls to spider for an IP. Saves CPU to getUrlFilterNum().
44rtmcmINT32robots.txt cache size50000000Bytes to use for caching robots.txt files for spidering per host.
45hpmcmINT32html pages cache size10000000Bytes to use for caching html pages for spidering per host.
46serpmcmINT32search results cache size10000000Bytes to use for caching search result pages per host.
47srcmaINT32search results cache max age10800How many seconds should we cache a search results page for?
48srtcpBOOL (0 or 1)send requests to compression proxy0If this is true, gb will route download requests for web pages to proxies in hosts.conf. Proxies will download and compress docs before sending back.
49pdnsIPdns 08.8.8.8IP address of the primary DNS server. Assumes UDP port 53. REQUIRED FOR SPIDERING! Use Google's public DNS 8.8.8.8 as default.
50sdnsIPdns 18.8.4.4IP address of the secondary DNS server. Assumes UDP port 53. Will be accessed in conjunction with the primary dns, so make sure this is always up. An ip of 0 means disabled. Google's secondary public DNS is 8.8.4.4.
51sdnsaIPdns 20.0.0.0All hosts send to these DNSes based on hash of the subdomain to try to split DNS load evenly.
52sdnsbIPdns 30.0.0.0
53sdnscIPdns 40.0.0.0
54sdnsdIPdns 50.0.0.0
55sdnseIPdns 60.0.0.0
56sdnsfIPdns 70.0.0.0
57sdnsgIPdns 80.0.0.0
58sdnshIPdns 90.0.0.0
59sdnsiIPdns 100.0.0.0
60sdnsjIPdns 110.0.0.0
61sdnskIPdns 120.0.0.0
62sdnslIPdns 130.0.0.0
63sdnsmIPdns 140.0.0.0
64sdnsnIPdns 150.0.0.0
65utBOOL (0 or 1)use threads1If enabled, Gigablast will use threads.
66utfdBOOL (0 or 1)use threads for disk1If enabled, Gigablast will use threads for disk ops. Now that Gigablast uses pthreads more effectively, leave this enabled for optimal performance in all cases.
67utfioBOOL (0 or 1)use threads for intersects and merges1If enabled, Gigablast will use threads for these ops. Default is now on in the event you have simultaneous queries so one query does not hold back the other. There seems to be a bug so leave this ON for now.
68utfscBOOL (0 or 1)use threads for system calls1Gigablast does not make too many system calls so leave this on in case the system call is slow.
69fwBOOL (0 or 1)flush disk writes1If enabled then all writes will be flushed to disk. If not enabled, then gb uses the Linux disk write cache and any hardware write cache. We seem to get better query performance when you enable this, but your disk writing operations will take longer.
70mbtfINT32min bytes to flush100000000Only call flush after we've written this many bytes on a particular file descriptor. A zero here means to flush after every disk write. Reducing flushing improves write performance. But flushing too little impacts disk read performance for queries as the kernel or hardware may only flush when its buffer is full, thereby causing spikey contention.
71vwlBOOL (0 or 1)verify written lists0Ensure lists being written to disk are not corrupt. That title recs appear valid, etc. Helps isolate sources of corruption. Used for debugging.
72cmbdBOOL (0 or 1)check memory before dumping0Check the in-memory structures before dumping to disk when they are full. This will SLOW DOWN your query response time during dumps, maybe even freezing things up for 10 seconds while checking. But this will detect BAD RAM DIMMS quite well.
73vdwBOOL (0 or 1)verify disk writes0Read what was written in a verification step. Decreases performance, but may help fight disk corruption mostly on Maxtors and Western Digitals.
74smdtINT32max spider read threads20Maximum number of threads to use per Gigablast process for accessing the disk for index-building purposes. Keep low to reduce impact on query response time. Increase for fast disks or when preferring build speed over lower query latencies
75sdtBOOL (0 or 1)separate disk reads1If enabled then we will not launch a low priority disk read or write while a high priority is outstanding. Help improve query response time at the expense of spider performance.



/admin/search - search controls   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionUse this collection. REQUIRED
4mtbtrINT64max bytes to read90000000Maximum number of bytes to read for a termlist per shard from a single file.
5msrpqINT32max search results per query100What is the limit to the total number of returned search results per query?
6msrINT32max search results in total200What is the maximum total number of returned search results.
7msrfpcINT32max search results in total for paying clients1000What is the limit to the total number of returned search results for clients.
8langweightFLOAT32language weight240.000000Defalt language weight if document matches query language, which can be provided using &qlang= or it the default can be set in the search controls. Use this to give results that match the specified the speicified &qlang higher ranking, or docs whose language is unknown. Can be override with &langw in the query url. Set to -1 to only return results in the query language, given by &qlang= or the default query language specified in the search controls. Setting to -1 is prefered over using the gblang: query operator because it is faster.
9mqtINT32max query terms999999Do not allow more than this many query terms. Helps prevent big queries from resource hogging.
10qedefBOOL (0 or 1)do query expansion by default1If enabled, query expansion will expand your query to include the various forms and synonyms of the query terms.
11spelldefBOOL (0 or 1)do spell checking by default0If enabled, when Gigablast finds a spelling recommendation it will be provided.
12autospellBOOL (0 or 1)auto correct spelling by default0If enabled, when Gigablast is CONFIDENT of a spelling recommendation it will automatically re-perform the query with the recommended spelling. Spell checking must be enabled for this to work. This is a default value and can be overriden directly with the autospell parm in each individual http request.
13defrelqueriesBOOL (0 or 1)show related queries by default0Offer related queries at bottom of search results.
14defquerycompBOOL (0 or 1)do query completion by default0Offer query completion suggestions as you type.
15saytBOOL (0 or 1)do search as you type0Update search results with each keystroke.
16qhBOOL (0 or 1)highlight query terms in summaries by default1Use to disable or enable highlighting of the query terms in the summaries.
17tmlINT32max title len80What is the maximum number of characters allowed in titles displayed in the search results?
18scdBOOL (0 or 1)site cluster by default0Should search results be site clustered? This limits each site to appearing at most twice in the search results. Sites are subdomains for the most part, like abc.xyz.com.
19hacrBOOL (0 or 1)hide all clustered results0Only display at most one result per site.
20drdBOOL (0 or 1)dedup results by default1Should duplicate search results be removed? This is based on a content hash of the entire document. So documents must be exactly the same for the most part.
21stgdblBOOL (0 or 1)do tagdb lookups for queries1For each search result a tagdb lookup is made, usually across the network on distributed clusters, to see if the URL's site has been manually banned in tagdb. If you don't manually ban sites then turn this off for extra speed.
22psdsINT32percent similar dedup summary default value90If document summary (and title) are this percent similar to a document summary above it, then remove it from the search results. 100 means only to remove if exactly the same. 0 means no summary deduping.
23msldINT32number of lines to use in summary to dedup4Sets the number of lines to generate for summary deduping. This is to help the deduping process not throw out valid summaries when normally displayed summaries are smaller values. Requires percent similar dedup summary to be non-zero.
24dduBOOL (0 or 1)dedup URLs by default0Should we dedup URLs with case insensitivity? This is mainly to correct duplicate wiki pages.
25defqlangSTRINGsort language preference defaultenDefault language to use for ranking results. Value should be any language abbreviation, for example "en" for English. Use xx to give ranking boosts to no language in particular. See the language abbreviations at the bottom of the url filters page.
26qcountrySTRINGsort country preference defaultusDefault country to use for ranking results. Value should be any country code abbreviation, for example "us" for United States. This is currently not working.
27smlINT32max summary len512What is the maximum number of characters displayed in a summary for a search result?
28smnlINT32max summary excerpts4What is the maximum number of excerpts displayed in the summary of a search result?
29smxcplINT32max summary excerpt length90What is the maximum number of characters allowed per summary excerpt?
30smwINT32max summary line width by default80<br> tags are inserted to keep the number of chars in the summary per line at or below this width. Also affects title. Strings without spaces that exceed this width are not split. Has no affect on xml or json feed, only works on html.
31clmfsINT32bytes of doc to scan for summary generation200000Truncating this will miss out on good summaries, but performance will increase.
32sfhtSTRINGfront highlight tagFront html tag used for highlightig query terms in the summaries displated in the search results.
33sbhtSTRINGback highlight tagFront html tag used for highlightig query terms in the summaries displated in the search results.
34dsrtINT32results to scan for gigabits generation by default30How many search results should we scan for gigabit (related topics) generation. Set this to zero to disable gigabits generation by default.
35iprBOOL (0 or 1)ip restriction for gigabits by default0Should Gigablast only get one document per IP domain and per domain for gigabits (related topics) generation?
36rotBOOL (0 or 1)remove overlapping topics1Should Gigablast remove overlapping topics (gigabits)?
37nrtINT32number of gigabits to show by default11What is the number of related topics (gigabits) displayed per query? Set to 0 to save CPU time.
38mtsINT32min gigabit score by default5Gigabits (related topics) with scores below this will be excluded. Scores range from 0% to over 100%.
39mdcINT32min gigabit doc count by default2How many documents must contain the gigabit (related topic) in order for it to be displayed.
40dspINT32dedup doc percent for gigabits (related topics)80If a document is this percent similar to another document with a higher score, then it will not contribute to the gigabit generation.
41mwptINT32max words per gigabit (related topic) by default6Maximum number of words a gigabit (related topic) can have. Affects xml feeds, too.
42tmssINT32gigabit max sample size4096Max chars to sample from each doc for gigabits (related topics).
43ddcBOOL (0 or 1)display dmoz categories in results1If enabled, results in dmoz will display their categories on the results page.
44didcBOOL (0 or 1)display indirect dmoz categories in results0If enabled, results in dmoz will display their indirect categories on the results page.
45dsclBOOL (0 or 1)display Search Category link to query category of result0If enabled, a link will appear next to each category on each result allowing the user to perform their query on that entire category.
46udfuBOOL (0 or 1)use dmoz for untitled1Yes to use DMOZ given title when a page is untitled but is in DMOZ.
47udsmBOOL (0 or 1)show dmoz summaries1Yes to always show DMOZ summaries with search results that are in DMOZ.
48sacotBOOL (0 or 1)show adult category on top0Yes to display the Adult category in the Top category
49hpSTRINGhome pageHtml to display for the home page. Leave empty for default home page. Use %N for total number of pages indexed. Use %n for number of pages indexed for the current collection. Use %c to insert the current collection name. Use %q to display the query in a text box. Use %t to display the directory TOP. Example to paste into textbox:
<html><title>My Gigablast Search Engine</title><script> function x(){document.f.q.focus();} </script><body onload="x()"><br><br><center><a href=/><img border=0 width=500 height=122 src=/logo-med.jpg></a><br><br><b>My Search Engine</b><br><br><form method=get action=/search name=f><input type=hidden name=c value="%c"><input name=q type=text size=60 value="">&nbsp;<input type="submit" value="Search"></form><br><center>Searching the <b>%c</b> collection of %n documents.</center><br></body></html>
50hhSTRINGhtml headHtml to display before the search results. Leave empty for default. Convenient for changing colors and displaying logos. Use the variable, %q, to represent the query to display in a text box. Use %e to print the url encoded query. Use %S to print sort by date or relevance link. Use %L to display the logo. Use %R to display radio buttons for site search. Use %F to begin the form. and use %H to insert hidden text boxes of parameters like the current search result page number. BOTH %F and %H are necessary for the html head, but do not duplicate them in the html tail. Use %f to display the family filter radio buttons. Example to paste into textbox:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <title>My Gigablast Search Results</title> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body%l> %F<table cellpadding="2" cellspacing="0" border="0"> <tr> <td valign=top>%L</td> <td valign=top> <nobr> <input type="text" name="q" size="60" value="%q"> %D<input type="submit" value="Blast It!" border="0"> </nobr> <br>%f %R </tr> </table> %H
51htSTRINGhtml tailHtml to display after the search results. Leave empty for default. Convenient for changing colors and displaying logos. Use the variable, %q, to represent the query to display in a text box. Use %e to print the url encoded query. Use %S to print sort by date or relevance link. Use %L to display the logo. Use %R to display radio buttons for site search. Use %F to begin the form. and use %H to insert hidden text boxes of parameters like the current search result page number. BOTH %F and %H are necessary for the html head, but do not duplicate them in the html tail. Use %f to display the family filter radio buttons. Example to paste into textbox:
<br> <table cellpadding=2 cellspacing=0 border=0> <tr><td></td> <td>%s</td> </tr> </table> Try your search on <a href=http://www.google.com/search?q=%e>google</a> &nbsp; <a href=http://search.yahoo.com/bin/search?p=%e>yahoo</a> &nbsp; <a href=http://search.dmoz.org/cgi-bin/search?search=%e>dmoz</a> &nbsp; </font></body>



/admin/spider - spider controls   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionUse this collection. REQUIRED
4cseBOOL (0 or 1)spidering enabled1Controls just the spiders for this collection.
5sitelistSTRINGsite listList of sites to spider, one per line. See example site list below. Gigablast uses the insitelist directive on the url filters page to make sure that the spider only indexes urls that match the site patterns you specify here, other than urls you add individually via the add urls or inject url tools. Limit list to 300MB. If you have a lot of INDIVIDUAL urls to add then consider using the addurl interface.
6restartUNARY CMD (set to 1)restart collectionRemove all documents from the collection and re-add seed urls from site list.
7mnsINT32max spiders300What is the maximum number of web pages the spider is allowed to download simultaneously PER HOST for THIS collection? The maximum number of spiders over all collections is controlled in the master controls.
8sdmsINT32spider delay in milliseconds0make each spider wait this many milliseconds before getting the ip and downloading the page.
9obeyRobotsBOOL (0 or 1)obey robots.txt1If this is true Gigablast will respect the robots.txt convention and rel no follow meta tags.
10usecloakingBOOL (0 or 1)use cloaking1Pretend our user-agent is googlebot for the purposes of parsing robots.txt. Does not actually set our user agent to googlebot.
11obeyRelNoFollowBOOL (0 or 1)obey rel no follow links1If this is true Gigablast will respect the rel no follow link attribute.
12mrcaINT32max robots.txt cache age86400How many seconds to cache a robots.txt file for. 86400 is 1 day. 0 means Gigablast will not read from the cache at all and will download the robots.txt before every page if robots.txt use is enabled above. However, if this is 0 then Gigablast will still store robots.txt files in the cache.
13cuaSTRINGuser agentIf empty then gigablast uses the default user agent specified in the master controls, otherwise, it uses this user agent for this collection only.
14useProxiesBOOL (0 or 1)always use spider proxies0If this is true Gigablast will ALWAYS use the proxies listed on the proxies page for spidering for this collection.
15automaticallyuseproxiesBOOL (0 or 1)automatically use spider proxies0Use the spider proxies listed on the proxies page if gb detects that a webserver is throttling the spiders. This way we can learn the webserver's spidering policy so that our spiders can be more polite. If no proxies are listed on the proxies page then this parameter will have no effect. When this is enabled gb will also use proxies for the sites listed in the sites to always use proxies text box on the proxies controls page which applies to all collections. It's basically a list of the sites we know are hard to spider without proxies.
16useeliteproxiesBOOL (0 or 1)use elite spider proxies0Use the elite spider proxies instead of the regular proxies.
17automaticallybackoffBOOL (0 or 1)automatically back off0Set the crawl delay to 5 seconds if gb detects that an IP is throttling or banning gigabot from crawling it. The crawl delay just applies to that IP. Such throttling will be logged.
18downloadiframesBOOL (0 or 1)expand iframes in spidered pages1Include iframes in the content of spidered pages.
19usetimeaxisBOOL (0 or 1)use time axis0If this is true Gigablast will index the same url multiple times if its content varies over time, rather than overwriting the older version in the index. Useful for archive web pages as they change over time.
20indexwarcsBOOL (0 or 1)index warc or arc files0If this is true Gigablast will index .warc and .arc files by injecting the pages contained in them as if they were spidered with the content in the .warc or .arc file. The spidered time will be taken from the archive file as well.
21dmtINT32daily merge time-1Do a tight merge on posdb and titledb at this time every day. This is expressed in MINUTES past midnight UTC. UTC is 5 hours ahead of EST and 7 hours ahead of MST. Leave this as -1 to NOT perform a daily merge. To merge at midnight EST use 60*5=300 and midnight MST use 60*7=420.
22dmdlSTRINGdaily merge days0Comma separated list of days to merge on. Use 0 for Sunday, 1 for Monday, ... 6 for Saturday. Leaving this parmaeter empty or without any numbers will make the daily merge happen every day
23dttBOOL (0 or 1)turing test enabled0If this is true, users will have to pass a simple Turing test to add a url. This prevents automated url submission.
24mauINT32max add urls0Maximum number of urls that can be submitted via the addurl interface, per IP domain, per 24 hour period. A value less than or equal to zero implies no limit.
25deBOOL (0 or 1)deduping enabled0When enabled, the spider will discard web pages which are identical to other web pages that are already in the index. However, root urls, urls that have no path, are never discarded. It most likely has to hit disk to do these checks so it does cause some slow down. Only use it if you need it.
26dewBOOL (0 or 1)deduping enabled for www1When enabled, the spider will discard web pages which, when a www is prepended to the page's url, result in a url already in the index.
27dcepBOOL (0 or 1)detect custom error pages1Detect and do not index pages which have a 200 status code, but are likely to be error pages.
28usrBOOL (0 or 1)use simplified redirects1If this is true, the spider, when a url redirects to a "simpler" url, will add that simpler url into the spider queue and abandon the spidering of the current url.
29ucrBOOL (0 or 1)use canonical redirects1If page has a on it then treat it as a redirect, add it to spiderdb for spidering and abandon the indexing of the current url.
30uimsBOOL (0 or 1)use ifModifiedSince0If this is true, the spider, when updating a web page that is already in the index, will not even download the whole page if it hasn't been updated since the last time Gigablast spidered it. This is primarily a bandwidth saving feature. It relies on the remote webserver's returned Last-Modified-Since field being accurate.
31mtotiINT32max term occurences to index300If a word is repeated too many times in the same document it can slow queries down quite a bit, so truncate each term at this many occurences. If you are doing book search then you might want to make this value higher.
32mlkftmINT32linkdb min files needed to trigger to merge6Merge is triggered when this many linkdb data files are on disk. Raise this when initially growing an index in order to keep merging down.
33mtftgmINT32tagdb min files to merge2Merge is triggered when this many linkdb data files are on disk.
34mtftmINT32titledb min files needed to trigger merge6Merge is triggered when this many titledb data files are on disk.
35mpftmINT32posdb min files needed to trigger to merge6Merge is triggered when this many posdb data files are on disk. Raise this while doing massive injections and not doing much querying. Then when done injecting keep this low to make queries fast.
36mtftspINT32spiderdb min files needed to trigger to merge4Merge is triggered when this many spiderdb data files are on disk.
37gltBOOL (0 or 1)enable link voting1If this is true Gigablast will index hyper-link text and use hyper-link structures to boost the quality of indexed documents. You can disable this when doing a ton of injections to keep things fast. Then do a posdb (index) rebuild after re-enabling this when you are done injecting. Or if you simply do not want link voting this will speed upyour injections and spidering a bit.
38csniBOOL (0 or 1)compute inlinks to sites1If this is true Gigablast will compute the number of site inlinks for the sites it indexes. This is a measure of the sites popularity and is used for ranking and some times spidering prioritzation. It will cache the site information in tagdb. The greater the number of inlinks, the longer the cached time, because the site is considered more stable. If this is NOT true then Gigablast will use the included file, sitelinks.txt, which stores the site inlinks of millions of the most popular sites. This is the fastest way. If you notice a lot of getting link info requests in the sockets table you may want to disable this parm.
39dlscBOOL (0 or 1)do link spam checking1If this is true, do not allow spammy inlinks to vote. This check is too aggressive for some collections, i.e. it does not allow pages with cgi in their urls to vote.
40ovpidBOOL (0 or 1)restrict link voting by ip1If this is true Gigablast will only allow one vote per the top 2 significant bytes of the IP address. Otherwise, multiple pages from the same top IP can contribute to the link text and link-based quality ratings of a particular URL. Furthermore, no votes will be accepted from IPs that have the same top 2 significant bytes as the IP of the page being indexed.
41uvfFLOAT32update link info frequency60.000000How often should Gigablast recompute the link info for a url. Also applies to getting the quality of a site or root url, which is based on the link info. In days. Can use decimals. 0 means to update the link info every time the url's content is re-indexed. If the content is not reindexed because it is unchanged then the link info will not be updated. When getting the link info or quality of the root url from an external cluster, Gigablast will tell the external cluster to recompute it if its age is this or higher.
42dsdBOOL (0 or 1)do serp detection1If this is eabled the spider will not allow any docs which are determined to be serps.
43mtdlINT32max text doc length1048576Gigablast will not download, index or store more than this many bytes of an HTML or text document. XML is NOT considered to be HTML or text, use the rule below to control the maximum length of an XML document. Use -1 for no max.
44modlINT32max other doc length1048576Gigablast will not download, index or store more than this many bytes of a non-html, non-text document. XML documents will be restricted to this length. Use -1 for no max.
45aftBOOL (0 or 1)apply filter to text pages0If this is false then the filter will not be used on html or text pages.
46filterSTRINGfilter nameProgram to spawn to filter all HTTP replies the spider receives, taking input as first arg (max length 64). Leave blank for none.
47ftoINT32filter timeout40Kill filter shell after this many seconds. Assume it stalled permanently.
48mitBOOL (0 or 1)make image thumbnails0Try to find the best image on each page and store it as a thumbnail for presenting in the search results.
49mtwhINT32max thumbnail width or height250This is in pixels and limits the size of the thumbnail. Gigablast tries to make at least the width or the height equal to this maximum, but, unless the thumbnail is sqaure, one side will be longer than the other.
50isrBOOL (0 or 1)index spider status documents0Index a spider status "document" for every url the spider attempts to spider. Search for them using special query operators like type:status or gberrorstr:success or stats:gberrornum to get a histogram. See syntax page for more examples. They will not otherwise show up in the search results.
51ahtsubhBOOL (0 or 1)assign spider host by url hash0Delegate spidering by url hash rather than by first ip. Good for collections which are only spidering one domain.
52ibBOOL (0 or 1)index body1Index the body of the documents so you can search it. Required for searching that. You wil pretty much always want to keep this enabled. Does not apply to JSON documents.
53hjxwfnBOOL (0 or 1)hash json or xml without field names0If this is enabled then you can search for test or cat when a JSON doc has "name":"cat" and it will still come up in the search results. Otherwise, you have to do a name:cat search. If this is set to NO it will save index space on disk.
54usubSTRINGurl find and replaceMatch all spidered urls to transform them. The format is space separated tuples: match_regex replace_val, one per line. If the replace val is omitted then it will be replaced with nothing. Back references are permitted using \1 - \9. For example:
\?.*
would remove all cgi parms.
(https?://)www\. \1mx\.
Would replace all www. sites with mx. sites.
55apiUrlSTRINGdiffbot api urlSend every spidered url to this url and index the reply in addition to the normal indexing process. Example: by specifying http://api.diffbot.com/v3/analyze?mode=high-precision&token= here you can index the structured JSON replies from diffbot for every url that is spidered. Gigablast will automatically append a &url= to this url before sending it to diffbot.
56urlProcessPatternTwoSTRINGdiffbot url process patternOnly send urls that match this simple substring pattern to Diffbot. Separate substrings with two pipe operators, ||. Leave empty for no restrictions.
57urlProcessRegExTwoSTRINGdiffbot url process regexOnly send urls that match this regular expression to Diffbot. Leave empty for no restrictions.
58pageProcessPatternTwoSTRINGdiffbot page process patternOnly send urls whose content matches this simple substring pattern to Diffbot. Separate substrings with two pipe operators, ||. Leave empty for no restrictions.



/admin/proxies - proxies   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3eliteproxyipsSTRINGelite spider proxy ipsLike spider proxies, but these are more expensive, so we don't send test queries.
4maxproxiestotryperipwhenconfidentINT32max proxies to try per ip (when confident of ban)10If we detect a proxy is banned how many more do we try before giving up. This is only when we are confident the proxy IP is banned. Like when we detect certain captchas. However, a lot of pages have captchas to prevent automated access and it does not mean your IP is banned.
5maxproxiestotryperppwhenunconfidentINT32max proxies to try per ip (when UNconfident of ban)2If we detect a proxy is banned how many more do we try before giving up. This is only when we are not so confident the proxy IP is banned, but it could be. Like, for instance, a TCP timedout or connection refused, or 403 Permission Denied response. All of which may or may not indicate a ban.
6minipcrawldelayperproxyMSINT32min ip crawl delay per proxy288000Each proxy waits at least this many milliseconds between downloads for urls from the same IP. Does not apply to super proxies (only 1 proxy of a given type specified).
7proxytesturlSTRINGspider proxy test urlhttp://www.gigablast.com/Download this url every minute through each proxy listed above to ensure they are up. Typically you should make this a URL you own so you do not aggravate another webmaster.
8proxytesttimeINT32spider proxy test ping time60Download the proxy test url once every this many seconds. Set to 0 to disable
9resetproxytableUNARY CMD (set to 1)reset proxy tableReset the proxy statistics in the table below. Makes all your proxies treated like new again.
10userandagentsBOOL (0 or 1)mix up user agents1Use random user-agents when downloading through a spider proxy listed above to protecting gb's anonymity. The User-Agent used is a function of the proxy IP/port and IP of the url being downloaded. That way it is consistent when downloading the same website through the same proxy.
11proxyAuthSTRINGsquid proxy authorized usersGigablast can also simulate a squid proxy, complete with caching. It will forward your request to the proxies you list above, if any. This list consists of space-separated username:password items. Leave this list empty to disable squid caching behaviour. The default cache size for this is 10MB per shard. Use item *:* to allow anyone access.
12alwaysuseproxysitesSTRINGsites to always use proxiesfacebook.com linkedin.com amazon.com yelp.com irs.gov quora.com stackexchange.comalways use proxies when hitting these sites. The collection must have always use proxies or automatically use proxies enabled before gb will use proxies for the listed sites.
13alwaysuseeliteproxysitesSTRINGsites to always use elite proxiesalways use elite proxies when hitting these sites. The collection must have always use proxies or automatically use proxies enabled before gb will use the elite proxies for the listed sites.



/admin/log - log controls   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3hrBOOL (0 or 1)log http requests1Log GET and POST requests received from the http server?
4laqBOOL (0 or 1)log autobanned queries1Should we log queries that are autobanned? They can really fill up the log.
5lqttINT32log query time threshold5000If query took this many millliseconds or longer, then log the query and the time it took to process.
6lqrBOOL (0 or 1)log query reply0Log query reply in proxy, but only for those queries above the time threshold above.
7lsuBOOL (0 or 1)log spidered urls1Log status of spidered or injected urls?
8lncBOOL (0 or 1)log network congestion0Log messages if Gigablast runs out of udp sockets?
9liBOOL (0 or 1)log informational messages1Log messages not related to an error condition, but meant more to give an idea of the state of the gigablast process. These can be useful when diagnosing problems.
10llBOOL (0 or 1)log limit breeches0Log it when document not added due to quota breech. Log it when url is too long and it gets truncated.
11ldaBOOL (0 or 1)log debug admin messages0Log various debug messages.
12ldbBOOL (0 or 1)log debug build messages0
13lddBOOL (0 or 1)log debug database messages0
14lddmBOOL (0 or 1)log debug dirty messages0
15lddiBOOL (0 or 1)log debug disk messages0
16ldpcBOOL (0 or 1)log debug disk page cache0
17lddnsBOOL (0 or 1)log debug dns messages0
18ldhBOOL (0 or 1)log debug http messages0
19ldiBOOL (0 or 1)log debug image messages0
20ldlBOOL (0 or 1)log debug loop messages0
21ldgBOOL (0 or 1)log debug language detection messages0
22ldliBOOL (0 or 1)log debug link info0
23ldmBOOL (0 or 1)log debug mem messages0
24ldmuBOOL (0 or 1)log debug mem usage messages0
25ldnBOOL (0 or 1)log debug net messages0
26ldqBOOL (0 or 1)log debug query messages0
27ldqcBOOL (0 or 1)log debug query completion messages0
28ldqtaBOOL (0 or 1)log debug quota messages0
29ldrBOOL (0 or 1)log debug robots messages0
30ldsBOOL (0 or 1)log debug spider cache messages0
31ldspBOOL (0 or 1)log debug speller messages0
32ldspsBOOL (0 or 1)log debug speller SPEED messages0
33ldsccBOOL (0 or 1)log debug sections messages0
34ldserpBOOL (0 or 1)log debug serp cache0
35ldsiBOOL (0 or 1)log debug seo insert messages0
36ldseoBOOL (0 or 1)log debug seo messages0
37ldstBOOL (0 or 1)log debug stats messages0
38ldsuBOOL (0 or 1)log debug summary messages0
39ldspidBOOL (0 or 1)log debug spider messages0
40ldspmthBOOL (0 or 1)log debug msg13 messages0
41dmthBOOL (0 or 1)disable host0 for msg13 reception hack0
42ldsprBOOL (0 or 1)log debug spider proxies0
43ldspuaBOOL (0 or 1)log debug url attempts0
44ldsdBOOL (0 or 1)log debug spider downloads0
45ldfbBOOL (0 or 1)log debug facebook0
46ldtmBOOL (0 or 1)log debug tagdb messages0
47ldtBOOL (0 or 1)log debug tcp messages0
48ldtbBOOL (0 or 1)log debug tcp buffer messages0
49ldthBOOL (0 or 1)log debug thread messages0
50ldtiBOOL (0 or 1)log debug title messages0
51ldtimBOOL (0 or 1)log debug timedb messages0
52ldtoBOOL (0 or 1)log debug topic messages0
53ldtopdBOOL (0 or 1)log debug topDoc messages0
54lduBOOL (0 or 1)log debug udp messages0
55ldunBOOL (0 or 1)log debug unicode messages0
56ldwlcBOOL (0 or 1)log debug winner list cache1
57ldreBOOL (0 or 1)log debug rebuild messages0
58ldpdBOOL (0 or 1)log debug pub date extraction messages0
59ltbBOOL (0 or 1)log timing messages for build0Log various timing related messages.
60ltadmBOOL (0 or 1)log timing messages for admin0Log various timing related messages.
61ltdBOOL (0 or 1)log timing messages for database0
62ltnBOOL (0 or 1)log timing messages for network layer0
63ltqBOOL (0 or 1)log timing messages for query0
64ltspcBOOL (0 or 1)log timing messages for spcache0Log various timing related messages.
65lttBOOL (0 or 1)log timing messages for related topics0
66lrBOOL (0 or 1)log reminder messages0Log reminders to the programmer. You do not need this.



/admin/masterpasswords - master passwords   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.



/admin/addcoll - add a new collection   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3addcollUNARY CMD (set to 1)add collectionAdd a new collection with this name. No spaces allowed or strange characters allowed. Max of 64 characters. REQUIRED



/admin/delcoll - delete a collection   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3delcollUNARY CMD (set to 1)delete collectionDelete the specified collection. You can specify multiple &delcoll= parms in a single request to delete multiple collections at once. REQUIRED



/admin/clonecoll - clone one collection's settings to another   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionClone settings INTO this collection. REQUIRED
4clonecollUNARY CMD (set to 1)clone collectionClone collection settings FROM this collection. REQUIRED



/admin/rebuild - rebuild data   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3rmeBOOL (0 or 1)rebuild mode enabled0If enabled, gigablast will rebuild the rdbs as specified by the parameters below. When a particular collection is in rebuild mode, it can not spider or merge titledb files.
4rctrSTRINGcollection to rebuildName of collection to rebuild. REQUIRED
5racBOOL (0 or 1)rebuild ALL collections0If enabled, gigablast will rebuild all collections.
6rmtuINT32memory to use for rebuild200000000In bytes.
7mrpsINT32max rebuild injections1Maximum number of outstanding injections for rebuild.
8rfrBOOL (0 or 1)full rebuild0If enabled, gigablast will reinject the content of all title recs into a secondary rdb system. That will the primary rdb system when complete.
9rfrknsxBOOL (0 or 1)add spiderdb recs of non indexed urls0If enabled, gigablast will add the spiderdb records of unindexed urls when doing the full rebuild or the spiderdb rebuild. Otherwise, only the indexed urls will get spiderdb records in spiderdb. This can be faster because Gigablast does not have to do an IP lookup on every url if its IP address is not in tagdb already.
10rrliBOOL (0 or 1)recycle link text1If enabled, gigablast will recycle the link text when rebuilding titledb. The siterank, which is determined by the number of inlinks to a site, is stored/cached in tagdb so that is a separate item. If you want to pick up new link text you will want to set this to NO and make sure to rebuild titledb, since that stores the link text. If you recycle link text the rebuild might be around twice as fast.
11rrtBOOL (0 or 1)rebuild titledb0If enabled, gigablast will rebuild this rdb
12rriBOOL (0 or 1)rebuild posdb1If enabled, gigablast will rebuild this rdb, ths index.
13rrclBOOL (0 or 1)rebuild clusterdb0If enabled, gigablast will rebuild this rdb
14rrspBOOL (0 or 1)rebuild spiderdb0If enabled, gigablast will rebuild this rdb
15rrldBOOL (0 or 1)rebuild linkdb0If enabled, gigablast will rebuild this rdb
16ruruBOOL (0 or 1)rebuild root urls1If disabled, gigablast will skip root urls.
17runruBOOL (0 or 1)rebuild non-root urls1If disabled, gigablast will skip non-root urls.



/admin/inject - inject url in the index here   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionInject into this collection. REQUIRED
4urlSTRINGurlSpecify the URL that will be immediately crawled and indexed in real time while you wait. The browser will return the final index status code. Alternatively, use the add url page to add urls individually or in bulk without having to wait for the pages to be actually indexed in realtime. By default, injected urls take precedence over the "insitelist" expression in the url filters so injected urls need not match the patterns in your site list. You can change that behavior in the url filters if you want. Injected urls will have a hopcount of 0. The injection api is described on the api page. Make up a fake url if you are injecting content that does not have one.

If the url ends in .warc or .arc or .warc.gz or .arc.gz Gigablast will index the contained documents as individual documents, using the appropriate dates and other meta information contained in the containing archive file. REQUIRED
5qtsSTRINGquery to scrapeScrape popular search engines for this query and inject their links. You are not required to supply the url parm if you supply this parm.
6injectlinksBOOL (0 or 1)inject links0Should we inject the links found in the injected content as well?
7spiderlinksBOOL (0 or 1)spider links0Add the outlinks of the injected content into spiderdb for spidering?
8newonlyBOOL (0 or 1)only inject content if new0If the specified url is already in the index then skip the injection.
9deleteurlBOOL (0 or 1)delete from index0Delete the specified url from the index.
10recycleBOOL (0 or 1)recycle content0If the url is already in the index, then do not re-download the content, just use the content that was stored in the cache from last time.
11dedupBOOL (0 or 1)dedup url0Do not index the url if there is already another url in the index with the same content.
12urlipIPurl IP0.0.0.0Use this IP when injecting the document. Do not use or set to 0.0.0.0, if unknown. If provided, it will save an IP lookup.
13hasmimeBOOL (0 or 1)content has mime0If the content of the url is provided below, does it begin with an HTTP mime header?
14delimSTRINGcontent delimeterIf the content of the url is provided below, then it consist of multiple documents separated by this delimeter. Each such item will be injected as an independent document. Some possible delimeters: ======== or <doc>. If you set hasmime above to true then Gigablast will check for a url after the delimeter and use that url as the injected url. Otherwise it will append numbers to the url you provide above.
15contenttypeSTRINGcontent typetext/htmlIf you supply content in the text box below without an HTTP mime header, then you need to enter the content type. Possible values: text/html text/plain text/xml application/json
16charsetINT32content charset106A number representing the charset of the content if provided below and no HTTP mime header is given. Defaults to utf8 which is 106. See iana_charset.h for the numeric values.
17contentSTRINGcontentIf you want to supply the URL's content rather than have Gigablast download it, then enter the content here. Enter MIME header first if "content has mime" is set to true above. Separate MIME from actual content with two returns. At least put a single space in here if you want to inject empty content, otherwise the content will be downloaded from the url. This is because the page injection form always submits the content text area even if it is empty, which should signify that the content should be downloaded.
18metadataSTRINGmetadataJson encoded metadata to be indexed with the document.
19sectionsBOOL (0 or 1)get sectiondb voting info0Return section information of injected content for the injected subdomain.
20diffbotreplySTRINGdiffbot replyUsed exclusively by diffbot. Do not use.



/admin/addurl - add url page for admin   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionAdd urls into this collection. REQUIRED
4urlsSTRINGurls to addList of urls to index. One per line or space separated. If your url does not index as you expect you can check it's spider history by doing a url: search on it. Added urls will have a hopcount of 0. Added urls will match the isaddurl directive on the url filters page. The add url api is described on the api page. REQUIRED
5stripBOOL (0 or 1)strip sessionids1Strip added urls of their session ids.
6spiderlinksBOOL (0 or 1)harvest links1Harvest links of added urls so we can spider them?.



/admin/reindex - query delete/reindex   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionquery reindex in this collection. REQUIRED
4qSTRINGquery to reindex or deleteWe either reindex or delete the search results of this query. Reindexing them will redownload them and possible update the siterank, which is based on the number of links to the site. This will add the url requests to the spider queue so ensure your spiders are enabled. REQUIRED
5srnINT32start result number0Starting with this result #. Starts at 0.
6ernINT32end result number99999999Ending with this result #. 0 is the first result #.
7qlangSTRINGquery languageenThe language the query is in. Used to rank results. Just use xx to indicate no language in particular. But you should use the same qlang value you used for doing the query if you want consistency.
8qrecycleBOOL (0 or 1)recycle content0If you check this box then Gigablast will not re-download the content, but use the content that was stored in the cache from last time. Useful for rebuilding the index to pick up new inlink text or fresher sitenuminlinks counts which influence ranking.
9forcedelBOOL (0 or 1)FORCE DELETE0Check this checkbox to delete the results, not just reindex them.



/admin/stats - general statistics   [ show parms in xml or json ]   [ show status in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.