Bots accessing Squash URLs

By default Squash will reject new deployment starts when the requests are coming from a known list of bots. This is helpful if you are not using Squash’s built-in Basic Authentication. We recommend enabling Basic Authentication though.

This only affects the deployment startup process. Squash will still allow bots to access environment URLs when they are already commissioned.

Here is a list of all the user agents  that are checked by Squash. Need to add more user agents? reach out to our support team.

Googlebot
Googlebot-Mobile
Googlebot-Image
Googlebot-News
Googlebot-Video
AdsBot-Google
AdsBot-Google-Mobile
Feedfetcher-Google
Mediapartners-Google
Mediapartners (Googlebot)
APIs-Google
bingbot
Slurp
[wW]get
curl
LinkedInBot
Python-urllib
python-requests
libwww
httpunit
nutch
Go-http-client
phpcrawl
msnbot
jyxobot
FAST-WebCrawler
FAST Enterprise Crawler
BIGLOTRON
Teoma
convera
seekbot
Gigabot
Gigablast
exabot
ia_archiver
GingerCrawler
webmon 
HTTrack
grub.org
UsineNouvelleCrawler
antibot
netresearchserver
speedy
fluffy
bibnum.bnf
findlink
msrbot
panscient
yacybot
AISearchBot
ips-agent
tagoobot
MJ12bot
woriobot
yanga
buzzbot
mlbot
YandexBot
yandex.com/bots
purebot
Linguee Bot
CyberPatrol
voilabot
Baiduspider
citeseerxbot
spbot
twengabot
postrank
turnitinbot
scribdbot
page2rss
sitebot
linkdex
Adidxbot
blekkobot
ezooms
dotbot
Mail.RU_Bot
discobot
heritrix
findthatfile
europarchive.org
NerdByNature.Bot
sistrix crawler
Ahrefs(Bot|SiteAudit)
fuelbot
CrunchBot
centurybot9
IndeedBot
mappydata
woobot
ZoominfoBot
PrivacyAwareBot
Multiviewbot
SWIMGBot
Grobbot
eright
Apercite
semanticbot
Aboundex
domaincrawler
wbsearchbot
summify
CCBot
edisterbot
seznambot
ec2linkfinder
gslfbot
aiHitBot
intelium_bot
facebookexternalhit
Yeti
RetrevoPageAnalyzer
lb-spider
Sogou
lssbot
careerbot
wotbox
wocbot
ichiro
DuckDuckBot
lssrocketcrawler
drupact
webcompanycrawler
acoonbot
openindexspider
gnam gnam spider
web-archive-net.com.bot
backlinkcrawler
coccoc
integromedb
content crawler spider
toplistbot
it2media-domain-crawler
ip-web-crawler.com
siteexplorer.info
elisabot
proximic
changedetection
arabot
WeSEE:Search
niki-bot
CrystalSemanticsBot
rogerbot
360Spider
psbot
InterfaxScanBot
CC Metadata Scaper
g00g1e.net
GrapeshotCrawler
urlappendbot
brainobot
fr-crawler
binlar
SimpleCrawler
Twitterbot
cXensebot
smtbot
bnf.fr_bot
A6-Indexer
ADmantX
Facebot
OrangeBot
memorybot
AdvBot
MegaIndex
SemanticScholarBot
ltx71
nerdybot
xovibot
BUbiNG
Qwantify
archive.org_bot
Applebot
TweetmemeBot
crawler4j
findxbot
S[eE][mM]rushBot
yoozBot
lipperhey
Y!J
Domain Re-Animator Bot
AddThis
Screaming Frog SEO Spider
MetaURI
Scrapy
Livelap[bB]ot
OpenHoseBot
CapsuleChecker
collection@infegy.com
IstellaBot
DeuSu
betaBot
Cliqzbot
MojeekBot
netEstate NE Crawler
SafeSearch microdata crawler
Gluten Free Crawler
Sonic
Sysomos
Trove
deadlinkchecker
Slack-ImgProxy
Embedly
RankActiveLinkBot
iskanie
SafeDNSBot
SkypeUriPreview
Veoozbot
Slackbot
redditbot
datagnionbot
Google-Adwords-Instant
adbeat_bot
WhatsApp
contxbot
pinterest
electricmonk
GarlikCrawler
BingPreview
vebidoobot
FemtosearchBot
Yahoo Link Preview
MetaJobBot
DomainStatsBot
mindUpBot
Daum
Jugendschutzprogramm-Crawler
Xenu Link Sleuth
Pcore-HTTP
moatbot
KosmioBot
pingdom
PhantomJS
Gowikibot
PiplBot
Discordbot
TelegramBot
Jetslide
newsharecounts
James BOT
Barkrowler
TinEye
SocialRankIOBot
trendictionbot
Ocarinabot
epicbot
Primalbot
DuckDuckGo-Favicons-Bot
GnowitNewsbot
Leikibot
LinkArchiver
YaK
PaperLiBot
Digg Deeper
dcrawl
Snacktory
AndersPinkBot
Fyrebot
EveryoneSocialBot
Mediatoolkitbot
Luminator-robots
ExtLinksBot
SurveyBot
NING
okhttp
Nuzzel
omgili
PocketParser
YisouSpider
um-LN
ToutiaoSpider
MuckRack
Jamie's Spider
AHC
NetcraftSurveyAgent
Laserlikebot
Apache-HttpClient
AppEngine-Google
Jetty
Upflow
Thinklab
Traackr.com
Twurly
Mastodon
http_get
DnyzBot
botify
007ac9 Crawler
BehloolBot
BrandVerity
check_http
BDCbot
ZumBot
EZID
ICC-Crawler
ArchiveBot
^LCC 
filterdb.iss.net/crawler
BLP_bbot
BomboraBot
Buck
Companybook-Crawler
Genieo
magpie-crawler
MeltwaterNews
Moreover
newspaper
ScoutJet
sentry
StorygizeBot
UptimeRobot
OutclicksBot
seoscanners
Hatena
Google Web Preview
MauiBot
AlphaBot
SBL-BOT
IAS crawler
adscanner
Netvibes
acapbot
Baidu-YunGuanCe
bitlybot
blogmuraBot
Bot.AraTurka.com
bot-pge.chlooe.com
BoxcarBot
BTWebClient
ContextAd Bot
Digincore bot
Disqus
Feedly
Fetch
Fever
Flamingo_SearchEngine
FlipboardProxy
g2reader-bot
imrbot
K7MLWCBot
Kemvibot
Landau-Media-Spider
linkapediabot
vkShare
Siteimprove.com
BLEXBot
DareBoost
ZuperlistBot
Miniflux
Feedspotbot
Diffbot
SEOkicks
tracemyfile
Nimbostratus-Bot
zgrab
PR-CY.RU
AdsTxtCrawler
Datafeedwatch
Zabbix
TangibleeBot
google-xrawler
axios
Amazon CloudFront