Monitoring Domains

Performing health checkups on domain names

Domain Health Checks

Ghostwriter grades a domain's health as Healthy or Burned. Health is reported as an overall health grade and a separate grade for the domain's DNS.

Categorization Health

Domain categories are pulled from multiple resources to be thorough. All of these services are checked:

  • VirusTotal

  • Cisco Talos

  • IBM Xforce

  • Fortiguard

  • Bluecoat

  • OpenDNS

  • Trendmicro

These checks use websites that sometimes deploy reCAPTCHAs when repeatedly hit with queries, so results may vary depending on how often your IP address hits these websites with requests.

Ghostwriter waits 20 seconds between requests to avoid slamming websites with requests and to accommodate VirusTotal's free API limit of four requests every 60 seconds.

These categories are bad and any source flagging a domain with one of these categories will trigger the health status to flip to Burned:

  • spam

  • phishing

  • gambling

  • suspicious

  • pornography

  • placeholders

  • web ads/analytics

  • scam/questionable/illegal

  • malicious sources/malnets

Most of these categories are self-explanatory, but some ⁠— like gambling ⁠— may not seem like they belong.

  • Placeholders: This often appears when a domain's category is undetermined. It translates to Uncategorized, and may mean the domain is under review.

  • Gambling: Not malicious, but likely blocked in a corporate environment.

If a domain is flagged as Burned it may still be recoverable. If you have a domain you really like, it may be worth trying to get it recategorized and continuing to monitor its reputation to determine if it can be used after a cool-off period.

Additional Checks

The infrastructure manager also references malwaredomains.com to check if one of your domain names appears in their list of malicious domains.

MX Toolbox is checked to determine if the domain name has been added to any spam or mailing blacklists, so you know if the domain name has gained a bad reputation.

DNS Health

The DNS health is based on VirusTotal's passive DNS report and checking to see if the IP addresses have appeared in any threat reports. If you bought an expired domain it's not at all strange to learn it once pointed at a cloud IP address that was flagged for something naughty at some point.

You will almost certainly see a Healthy domain with questionable DNS at some point. This is not something to be worried about without some human investigation.

Check to see if the IP addresses in question are yours. If they are not then you can probably ignore this. If the IP address was flagged very recently, like just before you bought the domain, then that may be a concern because the domain may be flagged for recent malicious activity. There's a lot of "maybes" here because this is very much an imperfect grade.

In general, focus on the overall health status (based on categories) and just use the passive DNS information and flags to help with manual analysis of your domains.

Manually Queuing a Monitoring Check

Scheduling these tasks will keep records up-to-date without requiring any user interaction.

Domain update (DNS and categorization) tasks exist in the tasks.py. These functions can be scheduled or requested manually.

Update All Domains
Update Individual Domains

The Domain Update Control Panel lives at /shepherd/update and provides information on when the updates were last run, how long they took to complete, and their exit state (success or error messages).

The Control Panel Under /shepherd/update

Click the Start Update button under the desired check to queue a check for all domains.

To update domain information or DNS records for just a single domain, open the domain's details and expand the Health and Categories or DNS Records panes.

Each of these panes contains a Refresh button. Click this button to queue an update for just the one domain.