Skip to content

Watchdog Installation Guide (with Browsermon Inspect, ucs, eti)

Introduction

This guide explains how to install and configure Watchdog using watchdog-installer.sh. Watchdog can optionally integrate with browsermon inspect (for data ingestion) as a Splunk replacement, and ETI and UCS for classifying URLs.

The installer supports: 1. Interactive prompts for Docker registry authentication (optional). 2. Enabling/disabling Kafka mode and/or UCS or ETI mode. 3. Automatic creation of necessary directories under /opt/watchdog. 4. File-by-file copy of important Watchdog files (prompts only for /opt/watchdog/watchdog/ overwrites). 5. Automatic generation of a .env file in your current directory, containing the environment variables Docker Compose will need. 6. A final Docker Compose deployment that launches the selected services.

Prerequisites

  1. Root/Sudo Access
    The installer must be run as root (or with sudo). It manages system directories (e.g., /opt/watchdog) and sets ownership of data directories.

  2. Docker and Docker Compose

    • Docker installed and running (docker ps should work).
    • Docker Compose plugin or Docker Compose CLI installed.
    • Optionally, Docker registry credentials if you plan to pull images from a private Docker registry.

  3. Local Files/Directories

    • A local deps/ directory that contains:
      • deps/connect-jars/ (Kafka connector JARs).
      • deps/watchdog/ (Watchdog source files).
      • deps/init-kafka-connect.sh (initialization script).
    • Docker Compose YAML files in the same directory from which you run the installer:
      • docker-compose.base.yml (required).
      • docker-compose.kafka.yml (if enabling Kafka).
      • docker-compose.elastic.yml (if enabling ETI or UCS).
      • docker-compose.eti.yml (if enabling ETI)
      • docker-compose.ucs.yml (if enabling UCS)
    • no-proxy.env — must be present alongside the Docker Compose files. This file clears host-level proxy variables inside containers to prevent them from breaking container-to-container networking.

Installation Steps

  1. Download the Watchdog release folder from Eunomatix.

Note: If you don't have internet access, first download the offline images and load them. See the "Offline Image Deployment (Optional)" section for more details.

  1. Extract the folder and make sure that watchdog-installer.sh is executable: 

    chmod +x watchdog-installer.sh

  2. Run the installer (as root): 

    sudo ./watchdog-installer.sh install

    • The script will:
      1. Prompt you for Docker registry authentication (optional).
      2. Prompt whether to enable Kafka/Elasticsearch modes.
      3. If Kafka mode is enabled, prompt for a KAFKA_EXTERNAL_IP.
      4. If Elasticsearch mode is enabled, prompt for host, port, passwords, etc.
      5. Create /opt/watchdog, /opt/watchdog/kafka_data, and /opt/watchdog/elasticsearch_data as needed.
      6. Copy files from deps/ into /opt/watchdog.
        • connect-jars and init-kafka-connect.sh are forced overwrites (no prompt).
        • The watchdog directory is copied file-by-file with a prompt for each existing file.
      7. Generate a .env file in your current directory (where Docker Compose can see it).
      8. Finally, run docker compose up -d using docker-compose.base.yml, plus the Kafka and/or Elastic Compose files if those modes were selected.

  3. Verify installation:

    • Check running containers: 
      docker ps
    • If Kafka was enabled:
      • kafka and kafka-connect containers should be running.
    • If Elasticsearch was enabled:
      • An elasticsearch container should be running (depending on your compose files).

Environment Variables and .env File

The script automatically writes environment variables to a .env file in the current working directory. Docker Compose will automatically load them. If Kafka/Elasticsearch is enabled, you’ll see lines like:

KAFKA_EXTERNAL_IP=your.machine.ip
ELASTIC_HOST=elasticsearch
ELASTIC_PORT=9200
ELASTIC_PASSWORD=BrowsermonElasticAdmin
ELASTIC_USER_PASSWORD=BrowsermonElasticUser
ELASTIC_SCHEME=https

You can modify these directly if needed (though re-running the script may overwrite them).

Service Configuration

  1. Kafka

    • Typically uses port 8092 (or whatever is in your docker-compose.kafka.yml).
    • Uses Kafka Connect to push data to MongoDB (or other sinks).
    • The init-kafka-connect.sh script is placed in /opt/watchdog, but you typically don’t need to run it manually unless your setup requires it.

  2. MongoDB

    • Often deployed alongside Kafka (depending on your docker-compose.kafka.yml).
    • The sink connector is configured to push Watchdog data to MongoDB.

  3. Grafana

    • Grafana is the interface to the Kafka and uses port 1514 with the username and password as admin.

  4. Elasticsearch

    • Typically listens on 9200 for HTTP/HTTPS calls.
    • The default scheme is https (from the script prompt) but can be changed if you have a custom ES config.
    • If using elasticsearch.yml, it should be placed in deps/watchdog or your custom location and referenced by docker-compose.elastic.yml.

Updating the Installation

If you are performing an upgrade or have installed Watchdog before and want to preserve configuration and data, you should go to the previous release directory from where you installed the older version and run: 

sudo ./watchdog-installer.sh clean
This will clean all the containers, networks, and volumes, and ask if you also want to clean the images. Then, run the installer again. It will ask you for the upgrade; proceed with it. If you want to replace any file, press y, otherwise skip the prompts.

If you re-run the installer and /opt/watchdog is detected, the script enters Update Mode.

Example: 

sudo ./watchdog-installer.sh install
If it sees an existing installation, you’ll be asked:

Existing installation detected at /opt/watchdog
Do you want to proceed with the update? (y/n)

  • Prompt you for Docker registry authentication (optional).
  • Prompt whether to enable Kafka/ETI/UCS modes.
  • If Kafka mode is enabled, prompt for a KAFKA_EXTERNAL_IP.
  • If ETI/UCS mode is enabled, prompt for username, passwords, etc.
  • Prompted for overwriting files inside /opt/watchdog/watchdog.
  • Other files (like init-kafka-connect.sh or connect-jars) are overwritten automatically.
  • The script will then re-run Docker Compose to update containers.

Note: If there is any change in conf files then you should press y when installer prompt you to replace that file.

Uninstalling / Cleaning Up

To stop and remove the Watchdog containers (Kafka/Elasticsearch included), run: 

sudo ./watchdog-installer.sh clean
This will: 1. Look for docker-compose.base.yml, docker-compose.kafka.yml, and docker-compose.elastic.yml in your current directory. 2. Run docker compose down -v with whichever files are found, removing containers and volumes.

Note: This does not delete /opt/watchdog or the data directories. If you want to remove them entirely, you can do so manually: 

sudo rm -rf /opt/watchdog

Troubleshooting

1. Checking Logs

View logs for a specific container: 

docker logs <container-name>
Examples: - docker logs kafka-connect - docker logs elasticsearch

Watchdog Logging

Watchdog and its supporting services generate several log files. Reviewing these logs can help in identifying the issue.

Log File Locations

  • Watchdog Logs
    Path: /opt/watchdog/watchdog_data/logs/watchdog.log
    Contains all Watchdog runtime messages, including server startup, requests, errors, and configuration-related issues.
    Use this log as your primary reference when troubleshooting Watchdog behavior.

  • Threat Collector Logs
    Path: /opt/watchdog/elasticsearch_data/logs/threat_collector.log
    Contains logs generated by the Threat Collector service, which downloads and ingests threat intelligence data into Elasticsearch.

  • UCS Client Logs
    Path: /opt/watchdog/elasticsearch_data/logs/ucs_client.log
    Contains logs for the UCS Client, which handles periodic sync with the UCS API and updates the local Elasticsearch index.

2. Verifying Kafka Connect

Inside the Kafka Connect container: 

docker exec -it kafka-connect /bin/bash
Then check connector status: 
curl -X GET http://kafka-connect:8083/connectors/mongo-sink-connector/status
A valid Mongo Sink Connector shows: 
{
  "name": "mongo-sink-connector",
  "connector": {
    "state": "RUNNING",
    "worker_id": "connect-worker-1"
  },
  "tasks": [
    {
      "id": 0,
      "state": "RUNNING",
      "worker_id": "connect-worker-1"
    }
  ],
  "type": "sink"
}

3. Checking Elasticsearch

If Elasticsearch is running with HTTPS and basic auth: 

curl -k -u elastic:BrowsermonElasticAdmin https://localhost:9200/_cluster/health
- -k ignores self-signed certificate errors. - Adjust the user/password as you configured them during installation prompts.

4. Internet Access

Important: For the functioning of the Elasticsearch-based URL classification, the following domains must be accessible from the network where your Watchdog is deployed. - ETI API: eti.eunomatix.com - UCS API: ucs.eunomatix.com

Offline Image Deployment (Optional)

If you have Docker images saved locally (e.g., .tar files) for offline deployment: 1. Load them: 

docker load -i your_offline_watchdog_image.tar

Note: The offline images from the Eunomatix has the following naming convention watchdog-{version}.img

  1. Skip Docker Hub Login during the script’s prompts.
  2. Ensure the Docker Compose files reference the images you loaded (matching tags).