github.com/prometheus/prometheus breaking changes

[CHANGE] API: The OTLP receiver endpoint can now be enabled using `--web.enable-otlp-receiver` instead of `--enable-feature=otlp-write-receiver`. #14894

[CHANGE] Disallow configuring AM with the v1 api. #13883

[CHANGE] Remote-write: default enable_http2 to false. #15219

[CHANGE] PromQL: Range selectors and the lookback delta are now left-open, i.e. a sample coinciding with the lower time limit is excluded rather than included. #13904

[CHANGE] `holt_winters` is now called `double_exponential_smoothing` and moves behind the [experimental-promql-functions feature flag](https://prometheus.io/docs/prometheus/latest/feature_flags/#experimental-promql-functions). #14930

[CHANGE] regexp `.` now matches all characters (performance improvement). #14505

[CHANGE] Logging: the format of log lines has changed a little, along with the adoption of Go's Structured Logging package. #14906

[CHANGE] Don't create extra `_created` timeseries if feature-flag `created-timestamp-zero-ingestion` is enabled. #14738

[CHANGE] Float literals and time durations being the same is now a stable fetaure. #15111

[CHANGE] UI: The old web UI has been replaced by a completely new one that is less cluttered and adds a few new features (PromLens-style tree view, better metrics explorer, "Explain" tab). However, it is still missing some features of the old UI (notably, exemplar display and heatmaps). To switch back to the old UI, you can use the feature flag `--enable-feature=old-ui` for the time being. #14872

[CHANGE] Agent mode has been promoted to stable. The feature flag `agent` has been removed. To run Prometheus in Agent mode, use the new `--agent` cmdline arg instead. #14747

[CHANGE] Remove deprecated `storage.tsdb.allow-overlapping-blocks`, `alertmanager.timeout`, and `storage.tsdb.retention` flags. #14640, #14643

[CHANGE] Set the `GOMAXPROCS` variable automatically to match the Linux CPU quota. Use `--no-auto-gomaxprocs` to disable it. The `auto-gomaxprocs` feature flag was removed. #15376

[CHANGE] Set the `GOMEMLIMIT` variable automatically to match the Linux container memory limit. Use `--no-auto-gomemlimit` to disable it. The `auto-gomemlimit` feature flag was removed. #15373

[CHANGE] Scraping: Remove implicit fallback to the Prometheus text format in case of invalid/missing Content-Type and fail the scrape instead. Add ability to specify a `fallback_scrape_protocol` in the scrape config. #15136

[CHANGE] Config: remove expand-external-labels flag, expand external labels env vars by default. #14657

[CHANGE] Prometheus will not add or remove port numbers from the target address. `no-default-scrape-port` feature flag removed. #14160

[CHANGE] Remove deprecated `remote-write-receiver`,`promql-at-modifier`, and `promql-negative-offset` feature flags. #13456, #14526

[CHANGE] Kubernetes SD: Remove support for `discovery.k8s.io/v1beta1` API version of EndpointSlice. This version is no longer served as of Kubernetes v1.25. #14365

[CHANGE] Kubernetes SD: Remove support for `networking.k8s.io/v1beta1` API version of Ingress. This version is no longer served as of Kubernetes v1.22. #14365

[CHANGE] UTF-8: Enable UTF-8 support by default. Prometheus now allows all UTF-8 characters in metric and label names. The corresponding `utf8-name` feature flag has been removed. #14705, #15258

[CHANGE] Console: Remove example files for the console feature. Users can continue using the console feature by supplying their own JavaScript and templates. #14807

[CHANGE] SD: Enable the new service discovery manager by default. This SD manager does not restart unchanged discoveries upon reloading. This makes reloads faster and reduces pressure on service discoveries' sources. The corresponding `new-service-discovery-manager` feature flag has been removed. #14770

[CHANGE] Scraping: config `scrape_classic_histograms` was renamed to `always_scrape_classic_histograms`. #15178

[ENHANCEMENT] Add metric for number of rule groups loaded. #5090

[CHANGE] Rollback Dockerfile to version at 2.5.0. Rollback of the breaking change introduced in 2.6.0. #5122

[FEATURE] Add subqueries to PromQL. #4831

[FEATURE] [EXPERIMENTAL] Add support for disk size based retention. Note that we don't consider the WAL size which could be significant and the time based retention policy also applies. #5109 prometheus/tsdb#343

[FEATURE] Add CORS origin flag. #5011

[ENHANCEMENT] Consul SD: Add tagged address to the discovery metadata. #5001

[ENHANCEMENT] Kubernetes SD: Add service external IP and external name to the discovery metadata. #4940

[ENHANCEMENT] Azure SD: Add support for Managed Identity authentication. #4590

[ENHANCEMENT] Azure SD: Add tenant and subscription IDs to the discovery metadata. #4969

[ENHANCEMENT] OpenStack SD: Add support for application credentials based authentication. #4968

[BUGFIX] Avoid duplicate tests for alert unit tests. #4964

[BUGFIX] Don't depend on given order when comparing samples in alert unit testing. #5049

[BUGFIX] Make sure the retention period doesn't overflow. #5112

[BUGFIX] Make sure the blocks don't get very large. #5112

[BUGFIX] Don't generate blocks with no samples. prometheus/tsdb#374

[BUGFIX] Reintroduce metric for WAL corruptions. prometheus/tsdb#473

[CHANGE] **breaking** TSDB: Changed WAL record format for the experimental native histograms. #11783

[FEATURE] Promtool: Add support of selecting timeseries for TSDB dump. #11872

[ENHANCEMENT] Agent: Native histogram support. #11842

[ENHANCEMENT] UI: Show individual scrape pools on /targets page. #11142

[ENHANCEMENT] TSDB/Scrape: Support FloatHistogram that represents buckets as float64 values. #11522 #11817 #11716

[ENHANCEMENT] TSDB: Add gauge histogram support. #11783 #11840 #11814

[ENHANCEMENT] Support native histograms in federation. #11830

[ENHANCEMENT] SD: Add VM size label to azure service discovery. #11650

[ENHANCEMENT] SD: Add container ID as a meta label for pod targets for Kubernetes. #11844

[ENHANCEMENT] Rules: Support native histograms in recording rules. #11838

[FEATURE] Add 'keep_firing_for' field to alerting rules. #11827

[FEATURE] Added experimental GRPC apis

[CHANGE] Admin and lifecycle APIs now disabled by default, can be re-enabled via flags

[FEATURE/CHANGE] Remote read can be configured to not read data which is available locally. This is enabled by default.

[CHANGE] New staleness behavior. Series now marked stale after target scrapes no longer return them, and soon after targets disappear from service discovery.

[CHANGE] Flags switched to using Kingpin, all flags are now --flagname rather than -flagname.

[FEATURE] Rules can be grouped now. Rules within a rule group are executed sequentially.

[ENHANCEMENT] Bump Consul HTTP client timeout to not match the Consul SD watch timeout.

[CHANGE] Rewritten exposition format parser with much higher performance. The Protobuf exposition format is no longer supported.

[ENHANCEMENT] Go-conntrack added to provide HTTP connection metrics.

[FEATURE] Add timestamp() function to PromQL.

[CHANGE] Completely rewritten storage layer, with WAL. This is not backwards compatible with 1.x storage, and many flags have changed/disappeared.

[CHANGE] Rules files use YAML syntax now. Conversion tool added to promtool.

[BUGFIX] Fix connection leak in Consul SD.

[CHANGE] Example console templates updated for new storage and metrics names. Examples other than node exporter and Prometheus removed.

[CHANGE] Removed `count_scalar`, `drop_common_labels` functions and `keep_common` modifier from PromQL.

[ENHANCEMENT] Remove remote read from the query path if no remote storage is configured.

[CHANGE] Rework Kubernetes SD.

[BUGFIX] Validate query end time is not before start time.

[ENHANCEMENT] Always show instance labels on target page.

[ENHANCEMENT] Add fuzzy search to `/graph` textarea.

[ENHANCEMENT] Add Azure SD metrics.

[ENHANCEMENT] Add EC2 SD metrics.

[FEATURE] Add GCE metadata as Prometheus meta labels.

[BUGFIX] Error on negative `-storage.staleness-delta`

[FEATURE] Add support for interpolating `target_label`.

[ENHANCEMENT] Round scraping timestamps to milliseconds in web UI

[BUGFIX] Fix date parsing in console template graphs

[FEATURE] Allow configuring multiple Alertmanagers

[FEATURE] Add server name to TLS configuration

[FEATURE] Add labels for all node addresses and discover node port if available in Kubernetes SD

[ENHANCEMENT] More meaningful configuration errors

[ENHANCEMENT] Make number of storage fingerprint locks configurable

[BUGFIX] Fix static console files in Docker images

[BUGFIX] Fix console JS XHR requests for IE11

[BUGFIX] Add missing path prefix in new status page

[CHANGE] Remove kubelet port config option in Kubernetes SD configuration

[CHANGE] Rename `target_groups` to `static_configs` in config files

[CHANGE] Rename `names` to `files` in file SD configuration

[ENHANCEMENT] Support time range on /api/v1/series endpoint

[CHANGE] Vector matching takes grouping labels from one-side

[FEATURE] Add `ignoring` modifier for binary operations

[BUGFIX] Fix issue of hanging target scrapes

[FEATURE] Add pod discovery to Kubernetes service discovery

[ENHANCEMENT] Partition status page into individual pages

[FEATURE] Add experimental Microsoft Azure service discovery

[BUGFIX] Return proper HTTP status codes on API errors

[FEATURE] Omitted time parameter in API query defaults to now

[ENHANCEMENT] Less frequent time series file truncation

[FEATURE] Multiple authentication options for EC2 discovery added

[FEATURE] Several meta labels for EC2 discovery added

[FEATURE] Allow full URLs in static target groups (used e.g. by the `blackbox_exporter`)

[FEATURE] Add Graphite remote-storage integration

[FEATURE] Create separate Kubernetes targets for services and their endpoints

[FEATURE] Add `clamp_{min,max}` functions to PromQL

[BUGFIX] Add config overflow validation for TLS configuration

[BUGFIX] Skip already watched Zookeeper nodes in serverset SD

[BUGFIX] Don't federate stale samples

[BUGFIX] Move NaN to end of result for `topk/bottomk/sort/sort_desc/min/max`

[BUGFIX] Limit extrapolation of `delta/rate/increase`

[BUGFIX] Fix unhandled error in rule evaluation

[ENHANCEMENT] Ignore lost+found directory during storage version detection

[CHANGE] "Healthy" and "unhealthy" targets are now called "up" and "down" in the web UI

[BUGFIX] Fix Kubernetes authentication configuration

[BUGFIX] Fix stripped OFFSET from in rule evaluation and display

[BUGFIX] Do not crash on failing Consul SD initialization

[BUGFIX] Revert changes to metric auto-completion

[ENHANCEMENT] Instrument number of manually deleted time series

[CHANGE] Remove undocumented 2nd argument of the `delta` function.

[CHANGE] Kubernetes `masters` renamed to `api_servers`

The `delta()` function had an undocumented optional second boolean argument

The default full refresh interval for the file-based service discovery has been

[CHANGE] Change default full refresh interval for the file-based service

[CHANGE] Change all regexes in the configuration to do full-string matches

Release tarballs now contain the built binaries in a nested directory.

The `hash_mod` relabeling action now uses MD5 hashes instead of FNV hashes to

In relabeling, parts of a source label that weren't matched by

Queries no longer interpolate between two data points. Instead, the resulting

Regular expressions supplied via the configuration are now anchored to match

Global labels are not appended upon storing time series anymore. Instead,

The names and units of metrics related to remote storage sample appends have

The experimental support for writing to InfluxDB has been updated to work

Escape sequences in double- and single-quoted string literals in rules or query

Support for filter operators between two scalar values (like `2 > 1`) will be

[CHANGE] Vendoring is now done via govendor instead of godep.

[CHANGE] Change web UI root page to show the graphing interface instead of

[CHANGE] Append global labels only when communicating with external systems

[CHANGE] For alert `SUMMARY`/`DESCRIPTION` template fields, cast the alert

[CHANGE] In relabeling, don't include unmatched source label parts in the

[CHANGE] Release tarballs now contain the binaries in a nested directory.

[CHANGE] Update InfluxDB write support to work with InfluxDB 0.9.x.

[CHANGE] Remove interpolation of vector values in queries.

[CHANGE] Renamed `global.labels` to `global.external_labels`.

The DNS-SD meta label `__meta_dns_srv_name` was renamed to `__meta_dns_name`

[CHANGE] Rename the DNS-SD meta label `__meta_dns_srv_name` to

[FEATURE] Added initial stable API version 1 under `/api/v1`,

[FEATURE] Allow configuring query parameters when scraping metrics endpoints.

[CHANGE] Change default storage directory to `data` in the current

The default scrape interval has been changed back from 1 minute to

[CHANGE] Restore default ScrapeInterval of 1 minute instead of 10 seconds.

The default storage directory has been changed from `/tmp/metrics`

[CHANGE] Update node disk console for new filesystem labels.

[FEATURE] Add Consul's `ServiceAddress`, `Address`, and `ServicePort` as

[FEATURE] Add `hashmod` relabel action to allow for horizontal

[FEATURE] Add `honor_labels` scrape configuration option to not

[FEATURE] Add basic federation support on `/federate`.

[FEATURE] Add optional `RUNBOOK` field to alert statements.

[FEATURE] Add pre-relabel target labels to status page.

[FEATURE] Add version information endpoint under `/version`.

Relative paths for rule files are now evaluated relative to the config file.

External reachability flags (`-web.*`) consolidated.

The `rule_checker` tool has been replaced by `promtool` with

The special `__meta_consul_tags` label in Consul service discovery

[CHANGE] Consolidate external reachability flags (`-web.*`)into one.

[CHANGE] Improve label matching performance and treat unset labels

[CHANGE] Resolve rule files relative to config file.

[CHANGE] Surround `__meta_consul_tags` value with tag separators.

[CHANGE] Remove `rule_checker` tool and add generic `promtool` CLI

Empty labels are now removed upon ingestion into the

[CHANGE] Deprecate `keeping_extra` modifier keyword, rename it to