Skip to content

Troubleshooting Guide

Troubleshooting Watchdog

This section provides some troubleshooting tips related to Watchdog.

Initial Checks

  1. Check Port Status If you are experiencing issues with the Watchdog server, the first step is to check the status of the port Watchdog is listening on. Follow these steps to troubleshoot:

    • Use netstat to Check Port Status 
      netstat -tuln | grep <watchdog_port>
    • Check if Watchdog is listening on the port and bind to 0.0.0.0

  2. Logs

    • Set the loglevel to DEBUG in watchdog.conf (see Certificate Management for where edits to release-tree config files take effect).
    • Watchdog runtime logs are written to /opt/watchdog/watchdog_data/logs/watchdog.log regardless of loglevel.

  3. Container Status

    • List running containers: 
      docker ps
    • Tail recent output from the Watchdog container: 
      docker logs --tail 200 watchdog
    • Live resource usage: 
      docker stats watchdog

Docker Troubleshooting

1. Checking Logs

View logs for a specific container: 

docker logs <container-name>
Examples: - docker logs kafka-connect - docker logs es

2. Verifying Kafka Connect

Inside the Kafka Connect container: 

docker exec -it kafka-connect /bin/bash
Then check connector status: 
curl -X GET http://kafka-connect:8083/connectors/mongo-sink-connector/status
A valid Mongo Sink Connector shows: 
{
  "name": "mongo-sink-connector",
  "connector": {
    "state": "RUNNING",
    "worker_id": "connect-worker-1"
  },
  "tasks": [
    {
      "id": 0,
      "state": "RUNNING",
      "worker_id": "connect-worker-1"
    }
  ],
  "type": "sink"
}

3. Checking Elasticsearch

If Elasticsearch is running with HTTPS and basic auth: 

curl -k -u elastic:BrowsermonElasticAdmin https://localhost:9200/_cluster/health
- -k ignores self-signed certificate errors. - Adjust the user/password as you configured them during installation prompts.

4. Internet Access

Important For the functioning of the Elasticsearch-based URL classification, the following domains must be accessible from the network where your watchdog is deployed.

For ETI:

  • ETI API: eti.eunomatix.com

For UCS (if ucs_updates=true in watchdog.conf):

  • UCS API : ucs.eunomatix.com

Common Issues and Solutions

Server Not Starting

  • Review your Watchdog Config. Ensure that you are using compatible version of Watchdog.
  • Things that can go wrong in the config file.
  • MODE: Set the mode to either 'local' or 'cloud' based on your deployment.
  • BMKEY: Provide the Base64-encoded license key.
  • AUTHCODE: Add the Base64-encoded authorization code.
  • BLACKLIST: Mode for watchdog-blacklist.conf. Must be guid or ip. An invalid value will fail Watchdog startup.
  • LOGDIR: Path to your log directory.
  • LOGLEVEL: Set the desired loglevel. Set it to DEBUG if the program doesn't work the intended way.
  • LIMIT: Enable the rate limiting.
  • RATE: Give the rate on which API is accessible.

  • ALLOWEDHOSTS: Lists of the Hosts that can access Watchdog.

  • WATCHDOG_PORT: Port on which watchdog will be run.

  • CERTFILE: Path to the TLS server certificate (default ./cert/server.crt).
  • KEYFILE: Path to the TLS private key (default ./cert/server.key).
  • SSLCONFIG: Path to ssl-config.ini used by init-pki.sh to generate the shared PKI.

Server Not Behaving As Expected

  • Review your SSL certificate is valid and not expired. Check expiry with sudo openssl x509 -in /opt/watchdog/pki/server.crt -noout -enddate. See Certificate Management to rotate it.
  • Ensure you are not exceeding the max controllers limit.

Controllers Receiving 403 Unexpectedly

  • Check whether the controller's GUID or source IP is listed in /opt/watchdog/watchdog/watchdog-blacklist.conf.
  • Confirm the blacklist= value in watchdog.conf matches the kind of entries in the blacklist file (guid vs ip). Entries that don't parse for the configured mode (e.g. UUIDs when blacklist=ip) are logged at ERROR level at startup and dropped — grep watchdog.log for Invalid <mode> entry in blacklist.

Watchdog Won't Start After Upgrade

  • If you upgraded from 4.0.x and kept your old watchdog.conf, the SSL paths will be wrong. Update certfile= to ./cert/server.crt and keyfile= to ./cert/server.key, or move your existing certs to those names.
  • The blacklist= field is now strict: values other than guid or ip cause startup to fail. If you previously had blacklist=None or a CSV file path, change it to guid or ip.
  • See Certificate Management for the new shared PKI layout.

Watchdog Logging

Watchdog and its supporting services generate several log files. Reviewing these logs can help in identifying the issue.

Log File Locations

  • Watchdog Logs
    Path: /opt/watchdog/watchdog_data/logs/watchdog.log
    Contains all Watchdog runtime messages, including server startup, requests, errors, and configuration-related issues.
    Use this log as your primary reference when troubleshooting Watchdog behavior.

  • Threat Collector Logs
    Path: /opt/watchdog/elasticsearch_data/logs/threat_collector.log
    Contains logs generated by the Threat Collector service, which downloads and ingests threat intelligence data into Elasticsearch.

  • UCS Client Logs
    Path: /opt/watchdog/elasticsearch_data/logs/ucs_client.log
    Contains logs for the UCS Client, which handles periodic sync with the UCS API and updates the local Elasticsearch index.

Contacting Support

A small troubleshooter script ships with the release at /opt/watchdog-release/troubleshooting/watchdog_ts.py. It bundles logs, container output, certificate info, and system details into a single zip you can attach to a support ticket.

Run it from anywhere on the VM:

sudo python3 /opt/watchdog-release/troubleshooting/watchdog_ts.py

The script uses the Python 3 standard library only — there is nothing to pip install. It produces watchdog_archive.zip in the current working directory. Send this file to support@eunomatix.com along with a description of the issue and the time it occurred.

For non-default paths (e.g., custom log location), see --help for override flags such as --logs-dir, --pki-dir, and --api-url.

What the Archive Contains

The troubleshooter collects operational diagnostics needed to debug the issue:

  • Watchdog runtime logs (/opt/watchdog/watchdog_data/logs/)
  • Threat collector and UCS client logs if those services are enabled
  • Output of docker ps -a, plus docker logs --tail 500 and docker inspect for each Watchdog container that exists
  • Public TLS certificate details from /opt/watchdog/pki/server.crt (subject, issuer, validity, full SAN list, public key)
  • System information (kernel, distribution, glibc version, hostname, IP)
  • Listening sockets (ss -tulpn)
  • API probe results from https://localhost:8900 (a check-license and a get-licenses request)

What is not collected: private keys, the contents of watchdog.conf (and therefore your license key, auth code, or Elasticsearch passwords), and any end-user browser activity payloads.

Note on customer-replaced certificates. If you replaced the bundled certificate with one issued by your own CA (see Certificate Management), the archive will include the public details of that certificate — your CA's distinguished name, the certificate's subject and SAN list (which may include internal hostnames or IP ranges), and the validity dates. The private key is never collected. Review the contents of cert_info.txt inside the archive before sending if your organization treats this metadata as sensitive.