InfluxDB

InfluxDB is a time series database written in Go. This page contains notes relating to InfluxDB 2.x. My older notes on InfluxDB 1.x has been moved.

Introduction

InfluxDB 2.x integrates the TICK stack (Telegraf, InfluxDB, Chronograf, Kapacitor) into one binary.

Migration considerations

Before you mgirate to InfluxDB 2.x, be aware of a few things:

1. You can't run InfluxQL. All your existing dashboards based on InfluxQL will need to be rewritten
2. InfluxQL is nice and might make more sense than the SQL-based language, but it's completely different and converting SQL to this functional-like syntax can be tricky. You should probably learn flux before you do the migration.
3. Your telegraf configs will need some tweaking. The database defaults to collectd and requires a token
4. Everything requires a token or authentication. If you use the line protocol with a shell script somewhere, you will need to update these scripts.

Installation and migration from InfluxDB 1.x

I had an existing InfluxDB 1.x dataset which I want to migrate to InfluxDB 2.x. The upgrade is automatically triggered by the official docker image if it detects 1.x data but no 2.x data. You must also mount /var/lib/influxdb2 as that will be where the migrated data as well as any new data will be written to.

I used the following docker-compose when I first did the migration.

services:
  influxdb:
    image: influxdb:2.0.7
    hostname: influxdb
    networks:
      - influxnet
      - traefik
    expose:
      - "8086"
    restart: always
    volumes:
      - /var/volumes/influx/influxdb:/var/lib/influxdb
      - /var/volumes/influx/influxdb2:/var/lib/influxdb2
      - /var/volumes/influx/config/influxdb.conf:/etc/influxdb/influxdb.conf
      - /var/volumes/influx/types.db:/usr/share/collectd/types.db
    environment:
      DOCKER_INFLUXDB_INIT_USERNAME: admin
      DOCKER_INFLUXDB_INIT_PASSWORD: AdminPassword1
      DOCKER_INFLUXDB_INIT_ORG: home
      DOCKER_INFLUXDB_INIT_BUCKET: initial
      DOCKER_INFLUXDB_INIT_MODE: upgrade

Once the migration is complete, you can remove the INIT environment variables and the old influxdb volume.

Flux

Flux is the new query language introduced in InfluxDB v2. Its syntax is basically a list of functions that you apply to the data.

See: https://www.sqlpac.com/en/documents/influxdb-flux-language-advanced-features.html

Cheat sheet

Task description	Query
Selecting data	from(bucket: "mybucket")
Select a time range	Relative: |> range(start: -1h) Absolute: |> range(start: -1h, stop: -10m) Time range: |> range(start: 2021-01-25T00:00:00Z, stop: 2021-01-29T23:59:00Z) When dealing with Grafana, use this for the time range window: |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
Filtering data	By tag. Filter on _measurement to filter by a specific measurement (Eg. temperature, account info, etc.): |> filter(fn: (r) => r["_measurement"] == "account") Filter on specific tag keys that your data is labeled with. (Eg. If account measurements are tagged with usernames, so I could filter with username == 'leo') |> filter(fn: (r) => r["username"] == "leo") By field: If you need a specific field by name. Eg. number of VMs by a user. |> filter(fn: (r) => r["_field"] == "vm_count") If you want to remove specific values out. Eg. vm_count less than 4. |> filter(fn: (r) => r["_value"] > 4)
Complex filters	Filters take in a lambda function. If you want to add additional conditions, you can either pipe multiple filters together (which ends up as a conditional AND) or rewrite the filter function. Eg: |> filter(fn: (r) => r["_measurement"] == "account" and r["_field"] == "vm_count") You can use regular expressions as well with filters. |> filter(fn: (r) => r.hostname =~ /^host[1-8]$/ and r.hostname !~ /^host[5-6]$/ )
Rename column	Rename columns. |> rename(columns: {new: "old"}) |> rename(columns: {_value: "average", _time: "when"})
Drop a column	Remove columns. |> drop(fn: (column) => column =~ /^_(start|stop|measurement)/)
Display data	Use yield() to return the data. If you need to rename the value, use something like yield(name: "mean").
Windowing data	|> aggregateWindow(every: 10m, fn: mean, column: "colname") Null values are created by default. To disable null values from being created, set createEmpty to false. |> aggregateWindow(every: 10m, fn: mean, createEmpty: false)
Pivot data	Pivoting can be used to make values into columns. This is useful if you have 2 measurement values (eg. CPU used and CPU available) which you want to turn into fields in order to do math on. |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
Doing math on data	Use the map function to compute columns. The function you pass in can also do math. Eg: |> map(fn: (r) => ({_value: r.memoryallocated / r.memorytotal})) The with operator updates a column if it exists or creates a new column in the output table. |> map(fn: (r) => ({r with _value: r.memoryallocated / r.memorytotal}))

Get list of fields

Useful for Grafana variables, where you want to have the user be able to select specific sets of fields.

from(bucket: "cloud")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "account")
  |> distinct(column: "name")
  |> keep(columns: ["_value"])
  |> group()

Tasks

Importing data from one bucket to another

I had Telegraf data written to a Telegraf database. However, InfluxDB 2.x uses the collectd bucket. To import old Telegraf data to the new bucket, you can use the following flux query.

from(bucket: "telegraf/autogen")
  |> range(start: -30d)
  |> to(bucket: "collectd/autogen")

However, I ran into this error: input field "free" on measurement "disk" is type integer, already exists as type float dropped. As a workaround, I made a filter which converted values to floats for this particular measurement:

from(bucket: "telegraf/autogen")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "disk")
  |> map(fn:(r) => ({ r with x: float(v: r.x) }))
  |> to(bucket: "collectd/autogen")

Create a new variable

Unlike in Grafana where variables are part of a dashboard, in InfluxDB, variables are its own entity which needs to be created separately. To do so, go to settings -> variables. A variable can be defined as a query, csv, or key-pair map. Some common variables from queries can be found at https://docs.influxdata.com/influxdb/v2.0/visualize-data/variables/common-variables/.

To create a new 'hosts' variable which contains all Telegraf hosts pushing metrics to InfluxDB, use this query:

import "influxdata/influxdb/schema"
schema.tagValues(bucket: "collectd/autogen", tag: "host")

When creating a dashboard, you can inject this hosts variable in the query editor or reference it from the 'v' key. That is, v.hosts.