The Future is Here
Raiting:
9

Google has a creative new way to kill SaaS startups


Back in the old days, when Google (or any of its poorly tuned AIs) wanted to kill your business, it usually denied you access to some of its services, and it worked. You've probably heard horror stories: Websites disappear from Google search and go into oblivion YouTube videos are demonetized , and creators lose a source of income Android applications disappear from Google Play and cannot contact their users APIs become more expensive or simply out of date Last but not least less important, personal analogue to all of the above: people lose access to their Gmail accounts and their entire digital lifeimage I swear I read the FAQ!
Everything goes according to one scenario. At first, businesses deliberately use Google services in such a way that they depend entirely on them for their survival. Then Google's automated behemoth does its job: It slightly changes the position of its ass on a leather chair the size of a planet and, without noticing it, destroys myriad of (relatively) small companies the size of an ant in the process. And finally, ant-sized companies are desperate to tell Google they are crushed, but can only communicate with an automated suggestion form. Sometimes an ant-sized CEO knows someone on Google because they were college buddies, or a CTO writes a post that somehow ends up on the front page of Hacker News. Then Google notices the problem and sometimes sees it as worthy of a solution, usually out of fear of the regulatory implications that an ant revolution could have. For this reason, conventional wisdom from ants dictates that you should not overly rely on Google services. And if you can manage to avoid this addiction, everything should be fine.image Such a flat blue surface with a cool red roof! So convenient!
What's new under the moon
In today's series "The Internet Isn't What It Used To Be," we'll talk about a new way Google can inadvertently overwhelm your startup. It does not require you to use Google services in any (intentional) way . Did you know that your domains can be blacklisted by Google for no particular reason, and that this blacklist is used not only by Google Chrome, but several other software and hardware vendors as well? Did you know that these other vendors sync the list with wildly changing timings and interpretations, so that solving any problems becomes extremely stressful and unpredictable? At the same time, the terms for processing complaints about the blacklist in Google are measured in weeks?image Now this is how your site or SaaS application looks like This "feature" of the blacklist is called Google Safe Browsing , and in the image above you can read the message that your site visitors see if one of the domains is included in the Safe Browsing database. Warning texts range from “Spoof site” to “Site contains malware” (complete list here ), but they are all set against an equally scary red background that tries to prevent the user from going to the site as much as possible. The first time we heard about the issue was due to a spike in complaints from customers who came across this red warning while trying to use our SaaS application. The second time around, we are better prepared, so we already have free time to write this post. InvGate is an IT SaaS platform powered by AWS. It serves over 1000 corporate clients and millions of end users. Firms use the product to manage tickets and requests from their users. Can you imagine the reaction of IT managers when all of a sudden their ticket system starts showing end users such ominous security warnings. When we first encountered this problem, we were desperate to understand what was going on and figure out how Google Safe Browsing (GSB) works, while tech support was trying to cope with the flow of requests from customers. We quickly realized that the URL to the Amazon Cloudfront CDN , from which we were serving static resources (CSS, Javascript, etc.), got into the GSB database, and this caused the entire application to crash for customers using this particular CDN. A quick look at the labeled system showed that everything looks fine. While the Devops team was working in a rush to set up a new CDN and prepare to move customers to a new domain, I discovered in Google documentation that Google Search Console (GSC) can provide additional explanations about why a site is flagged in the database. ... I won't bore you with details, but in order to access this information, you must prove that you own the site , to do this, set up a custom DNS record or upload some files to the domain root. We did it - and in 20 minutes we received a report about our site. The report looked something like this:image This is ... not particularly informative. The report also had a Request Review button that I quickly clicked without actually taking any action on the site as there was no information about the alleged issue. I submitted a claim with the note that I did not have malicious URLs listed, although the documentation says that Google always provides sample URLs to help webmasters identify problems.image Fine! A request to check an invalid report may cause future checks to become even slower. About an hour later, the site was removed from the GSB database, we didn't even finish withdrawing clients from this CDN. About two hours later, an automatic email arrived confirming that the verification was successful. No clarification as to what caused the problem.
What happened next
During the week following the incident, we continued to receive periodic reports of access issues from customers. Google Safe Browsing provides two different APIs for use in commercial and non-commercial products. In our case, the problem was reproducible in at least some Firefox users, as well as in some antivirus and network security devices. They tagged our site and blocked access to it many days later . We continued to migrate all clients from the old CDN to the new one, and so we ended up fixing the problem forever. We never really found out the reason and blamed it on some stoned AI at Google headquarters.
How to prevent Google Safe Browsing from tagging your site
If you run a SaaS business and promise customers guaranteed availability, then listing in Google Safe Browsing for no specific reason is a very real business risk. Unfortunately, given the purely Google opaque mechanism for tagging and viewing sites, this is unlikely to be guaranteed to be avoided. But you can certainly design your application and processes to minimize the chances of this happening, reduce the impact of actual blacklisting, and minimize the time it takes to resolve the issue. Here are the steps we take ourselves that I can recommend: Don't keep all your eggs on the same domain . Apparently GSB tags entire domains or subdomains. Therefore, it is better to distribute applications across multiple domains, as this will reduce the damage from losing any of them. For example, company.сom for the site, app.company.net for the application, eucdn.company.nеt for customers in Europe, useastcdn.company.nеt for customers in the US East Coast, etc. Do not host customer data on primary domains . Domains are often blacklisted because SaaS clients unknowingly uploaded malicious files to the server. These files are harmless to systems, but their very existence can lead to the fact that the entire domain will be blacklisted. Anything your users upload to apps must be hosted outside of the main domains. For example, use companyusercontent.cоm to store client files. Actively claim domain ownership in Google Search Console . This will not prevent the site from being blacklisted, but you will receive an email that will allow you to quickly respond to the problem. It takes some time to respond to such incidents, and this is precious time for your customers to suffer. Be prepared to change your domain if needed . This is the hardest thing to do, but it is the only effective anti-blacklisting tool: design systems so that service domain names can be easily changed (via scripts or orchestration tools). For example, suppose eucdn.company2.net has a CNAME record for eucdn.company.net, and if the former is blocked, update the application configuration to load resources from an alternate domain.
What to do if your SaaS application or website is blacklisted by Google Safe Browsing
Here's what I would recommend: If you can easily and quickly switch your application to another domain, this is the only way to reliably, quickly and supposedly resolve the incident . If you can, do so. That's all. Otherwise, once you identify a blocked domain, review the reports in Google Search Console. If you still have not claimed ownership of the domain, you will have to do it right now, which will take some time. If the site is indeed hacked, fix the problem (for example, remove offensive content or hacked pages) and then request a security check. If the site is not hacked or the Safe Browsing report is meaningless, request a security check anyway and state that the report is incomplete. Then, instead of rushing about in agony, presenting the amount of damage for the waiting time, proceed with the transition to the new domain anyway. Verification may take several weeks.
Cherry on the cake
Several months after the first incident, we received an email from Search Console informing us that one of our domains was blacklisted again. A few hours later, as a G Suite domain administrator, I received another interesting email, which you can read below.image The "sc" in [email protected] stands for Search Console. Let me explain in my own words what it is, because it's just mind-blowing. This is an email warning from Search Console about being blacklisted. This second email says that the G Suite automatic phishing email filter considers this email from Google Search Console to be bogus . Of course, this is not the case , since our domain was indeed blacklisted. So Google ca n't even decide if its own phishing alerts are phishing . (Lol?)
Some unpleasant thoughts about the future of the internet
To anyone in the tech industry, it's pretty clear that the big tech giants are pretty much the guardians of the Internet. But before, I interpreted it in a free, metaphorical sense. The Safe Browsing incident described here made it very clear that Google literally controls who can access your site, no matter where or how you operate it. Since Chrome has about 70% of the market, and Firefox and Safari to some extent use the GSB database, Google can make any site virtually inaccessible on the Internet with one flick of a finger.
Skull 12 february 2021, 21:17
Vote for this post
Bring it to the Main Page
 

Comments

Leave a Reply

B
I
U
S
Help
Avaible tags
  • <b>...</b>highlighting important text on the page in bold
  • <i>..</i>highlighting important text on the page in italic
  • <u>...</u>allocated with tag <u> text shownas underlined
  • <s>...</s>allocated with tag <s> text shown as strikethrough
  • <sup>...</sup>, <sub>...</sub>text in the tag <sup> appears as a superscript, <sub> - subscript
  • <blockquote>...</blockquote>For  highlight citation, use the tag <blockquote>
  • <code lang="lang">...</code>highlighting the program code (supported by bash, cpp, cs, css, xml, html, java, javascript, lisp, lua, php, perl, python, ruby, sql, scala, text)
  • <a href="http://...">...</a>link, specify the desired Internet address in the href attribute
  • <img src="http://..." alt="text" />specify the full path of image in the src attribute