Using Telegraf input for influxdb 2 [closed]

Question

I have an external productive influxdb v2 OSS, I cannot alter or change. There is a second Influxdb 2 I use for analytics, so I try to sync the databases by ingesting from the first and outgesting to the second one using telegraf

For influxdb v1 I there was the default way to use telegraf on a system with [[inputs.influxdb]] and [[outputs.influxdb]].

But for influxdb v2 I only have an analog [[outputs.influxdb_v2]] but no direct input. How can I achieve the behavior with telegraf? Is there a possibility with other inputs to reuse for that?

Answer 1

InfluxDB has its own replication mechanism - and it is easy enough to use.

You only need to add a remote config on the source instance via

influx remote create -h

and then initialize the replication between the source ("main") bucket and the replica:

influx replication create -h

As usual, the devil is in the details, so I have made docker-compose.yaml and the other files needed available on GitHub.

The docker-compose.yaml

Basically this only sets up a "main" InfluxDB instance, a replication target ("replica") and a Telegraf to fill the bucket we want to replicate with some data.

x-common:
  # This is a common configuration for both the master and the replica.
  containerconfig: &containerconfig
    image: influxdb:2-alpine
    restart: always
    secrets:
      - influxdb2_admin_password
      - influxdb2_admin_token
    healthcheck:
      test: ["CMD","influx", "ping"]
      interval: 20s
      timeout: 3s
      retries: 5
      start_period: 10s
    environment: &common-env
      DOCKER_INFLUXDB_INIT_MODE: setup
      DOCKER_INFLUXDB_INIT_USERNAME: admin
      DOCKER_INFLUXDB_INIT_PASSWORD_FILE: /run/secrets/influxdb2_admin_password
      DOCKER_INFLUXDB_INIT_ADMIN_TOKEN_FILE: /run/secrets/influxdb2_admin_token
      DOCKER_INFLUXDB_INIT_ORG: &org yourorg
secrets:
  influxdb2_admin_password:
    file: ./influxdb2-admin-password
  influxdb2_admin_token:
    file: ./influxdb2-admin-token
volumes:
  replica-data:
    driver: local
  main-data:
    driver: local
  replica-config:
    driver: local
  main-config:
    driver: local
configs:
  telegraf:
    file: ./telegraf.conf
services:
  replica:
    << : *containerconfig
    ports:
      # We willl use this port to access the data
      # which will be replicated from the "maindb"
      # instance to "replicadb".
      - 8086:8086
    environment:
      << : *common-env
      DOCKER_INFLUXDB_INIT_BUCKET: yourdest
    volumes:
      - type: volume
        source: replica-data
        target: /var/lib/influxdb2
      - type: volume
        source: replica-config
        target: /etc/influxdb2
  main:
    << : *containerconfig
    depends_on:
      # We want to ensure that we can actually initialize
      # the replication
      replica:
        condition: service_healthy
    environment:
      << : *common-env
      DOCKER_INFLUXDB_INIT_BUCKET: yoursource
      DEST_BUCKET: yourdest
      REPLNAME: yourreplication
      REMOTE_URL: http://replica:8086
    volumes:
      - type: volume
        source: main-data
        target: /var/lib/influxdb2
      - type: volume
        source: main-config
        target: /etc/influxdb2
      - type: bind
        source: ./docker-entrypoint-initdb.d/setup-replication.sh
        target: /docker-entrypoint-initdb.d/setup-replication.sh
  telegraf:
    # Just to fill the "yoursourrce" bucket with some data
    # and to test the replication.
    image: telegraf:alpine
    user: telegraf
    depends_on:
      main:
        condition: service_healthy
    configs:
      - source: telegraf
        target: /etc/telegraf/telegraf.conf
    secrets:
      - influxdb2_admin_token
    environment:
      INFLUXDB_ORG: *org
      INFLUXDB_BUCKET: yoursource

docker-entrypoint-initdb.d/setup-replication.sh

This script sets up the actual replication. It is automatically executed when the "main" InfluxDB instance is started.

#!/bin/ash

set -e

# Extract the required variables from the environment
ADMIN_TOKEN=$(cat $DOCKER_INFLUXDB_INIT_ADMIN_TOKEN_FILE)
REMOTE_ORG_ID=$(influx org ls --host $REMOTE_URL -t $ADMIN_TOKEN -n $DOCKER_INFLUXDB_INIT_ORG  | awk -v pattern=$DOCKER_INFLUXDB_INIT_ORG '$0 ~ pattern {print $1}')
LOCAL_BUCKET_ID=$(influx bucket ls -t $ADMIN_TOKEN | awk -v pattern=$DOCKER_INFLUXDB_INIT_BUCKET '$0 ~ pattern {print $1}')

# Create a remote on the "source" machine, pointing to the replica.
influx remote create \
  --name replica \
  -o $DOCKER_INFLUXDB_INIT_ORG \
  -t $ADMIN_TOKEN \
  --remote-url $REMOTE_URL \
  --remote-api-token $ADMIN_TOKEN \
  --remote-org-id $REMOTE_ORG_ID --allow-insecure-tls

REMOTE_ID=$(influx remote list -t $ADMIN_TOKEN | awk -v pattern=$REMOTE_URL '$0 ~ pattern {print $1}')

# Create the replication to the source machine.
influx replication create \
  --name $REPLNAME \
  --remote-id $REMOTE_ID \
  --local-bucket-id $LOCAL_BUCKET_ID \
  --remote-bucket $DEST_BUCKET

The other files are not that important to understand what is going on.