At Coinbase, we use Datadog to collect system and application metrics, implement SLIs and SLOs, create dashboards, and more. As the number of dashboards and monitors grew, we began to see the need to codify them. We were worried that we didn’t have tools to detect accidental or malicious modification. Imagine a production incident that was not noticed by engineers because of an accidentally muted monitor.

Codification solves this because modifications are explicit (through code) and are stored in version control (benefitting from notification and code review systems).

One way of solving this problem would be to store the Datadog components (e.g. Datadog timeboards, screenboards, and monitors) in a version control system and apply changes done through code. The downside of this approach is that managing screenboards or timeboards though the code is not convenient or friendly.

Another way of solving this problem would be to use a Datadog UI-driven approach. This would be much more convenient and friendlier to use. This would require detecting changes, creating a request to submit it back to a version control system and the ability to revert/rollback changes that were not approved.

To achieve the best of both potential solutions (code and UI-driven approaches) we first started by reviewing existing projects we could find:

Each project had their pros and cons but this came down to two issues: not being able to codify dashboards in addition to monitors (even if we forked and contributed back) or being too complex for our customers.

Coinbase Watchdog is a GitHub app and a Golang service that uses the Datadog API to watch for changes in Datadog, achieving the best of both a code and UI-driven approach. When it sees a change, it automatically creates a Pull Request (PR) with the changes in a dedicated Datadog GitHub repository. We have control and consensus mechanisms (you can read more about Heimdall here) that provide us the guarantees that a sufficient number of people have reviewed the change before it can land on master. If a PR was not approved and closed by a customer, Watchdog will call Datadog APIs to restore the components from the master branch in source control.

This approach gives us a UI-driven codification bot. All changes made in the Datadog UI will be automatically picked up by the bot and corresponding Pull Requests will be created. Watchdog can also detect if a user modified the code (Datadog component JSON) and apply the change to Datadog.

The two workflows are pictured here:

The Watchdog service has two types of configuration:

This is an example of a User configuration YAML file:

The components that you see above are:

The Watchdog service is completely stateless. There are two ways in which Watchdog detects changes: Full sync and Incremental.

When the Watchdog service starts for the first time, it will query all components by ID and check each against the components stored in GitHub. If some component files are different, new PRs will be created per user configuration file. Depending on users’ configuration files, this step could potentially make many Datadog API calls. However, this will only happen once on service startup to verify that the current state in Datadog is consistent with the source in GitHub.

This is a fairly straight-forward task, and consists of several steps:

An example API response from the Datadog API showing the “modified” key:

When Watchdog creates a new PR it is important to notify the relevant team so that they can review the PR. We use a GitHub’s CODEOWNERS feature for that. In the GitHub repository root we have a CODEOWNERS file with the following lines:

If a PR affects files in config/reliability/* or data/infra/reliability/* the Reliability Engineering team will be notified by email.

Furthermore, a team can opt-in to Slack notifications by setting a “slack” field under “meta” in user config file with a slack channel (see above picture).

In the future, we would like to add more features and are already working on a way to automatically revert changes based on PR expiration. We’re also looking forward to see if others who use Datadog find this service useful and are able and willing to contribute.

Datadog introduced new dashboard APIs which is currently not supported by Watchdog. We are planning to add this feature soon and meanwhile PRs are highly appreciated ;)

If you’re interested in contributing to this project, check it out on GitHub here!

If you’re interested in helping us build a modern, scalable platform for the future of crypto markets, we’re hiring in San Francisco!

This website may contain links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

Unless otherwise noted, all images provided herein are by Coinbase.

Introducing Coinbase Watchdog: A new approach to UI and code-driven automation for Datadog was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Similar to Notcoin - Blum - Airdrops In 2024

origin »

54 +