Skip to main content

DataDios Workload Analyzer

Overview​

DataDios Workload Analyzer is a powerful tool designed to analyze and optimize workloads across various cloud and on-prem platforms like databases, data governance tools, etc. It provides insights into resource utilization, performance metrics, and cost efficiency, enabling organizations to make informed decisions about their cloud and on-prem infrastructure.

Key Features​

Comprehensive Workload Analysis​

Workload Analyzer offers a detailed analysis of workloads, including CPU, storage, and cost in some cases. It also helps:

  • Identify underutilized resources
  • Identify potential bottlenecks in performance
  • Fine-tuning work units and utilization

Considering a database as a data source for workload analysis, the tool can help with:

  • analyzing query performance
  • indexing efficiency
  • long executing queries
  • IO and storage usage
  • Identifying relations between queries and underlying tables / views.

Multi-Cloud Support​

The tool supports multiple cloud platforms, including AWS, IBM Watsonx, IBM WKC, data governance tools and private clouds. This ensures that organizations can analyze workloads across their entire cloud infrastructure.

Supported Data Source Types​

Workload Analyzer can run against the following data source types:

  • Oracle Database
  • Snowflake Database
  • PostgreSQL Database
  • Watsonx.data
  • Trino Engine
  • MySQL Database
  • OpenAI Model
  • Ollama Model
  • AWS Service
  • Hetzner Cloud
  • Azure Service
  • Remote Host
  • Cost Dashboard
  • DataDios
  • SQLServer Database
  • Airflow
  • SmartDiff Dashboard
  • Redis
  • ArcadeDB Database
  • GitHub

Redis Metrics​

A Redis data source can collect Redis Metrics (turn it on under the data source's workload metric groups). A sample is taken every 5 minutes and shown as five charts:

  • Used Memory (MB)
  • Memory Used (% of maxmemory): blank when Redis has no maxmemory limit, because there is no limit to be a percentage of.
  • Evicted Keys: keys evicted in each 5-minute interval, not the total since Redis last restarted.
  • Keyspace Hit Rate (%): hits as a share of lookups in each interval. Blank for an interval with no lookups.
  • Connected Clients

Put the Redis password in the data source's password field, which is stored encrypted. The first sample after the metric group is turned on shows Redis's totals since its last restart; every sample after that shows one interval.

ArcadeDB Health​

An ArcadeDB Database data source can collect ArcadeDB Health, sampled every 5 minutes:

  • Pgwire :5432 Up: 1 when ArcadeDB's PostgreSQL protocol answers a query, 0 when it does not. Set pgwire_port on the data source if it is not 5432.
  • Database Size (GB)
  • Pgwire Query Latency (ms): blank when pgwire is down.

If ArcadeDB cannot be reached at all, no sample is recorded and the collection job fails; a scheduler alert on that job catches it.

GitHub CI Runs​

A GitHub data source can collect GitHub CI Runs (turn it on under the data source's workload metric groups). Every hour it counts the GitHub Actions workflow runs started in that hour, per repository and workflow, and shows three charts, one line per repository:

  • CI Failure Rate (%): failed runs as a share of finished runs (successful plus failed). Blank for an hour with no finished runs.
  • Failed Runs: failed runs per hour.
  • Total Runs: all runs per hour.

What is counted:

  • Failed means the run ended as failure, timed_out or startup_failure. Runs that were cancelled are counted separately and stay out of the failure rate. Runs that ended as skipped, neutral, action_required or stale are counted as other: they are in the total but in neither side of the failure rate.
  • A run still queued or running when its hour is collected is counted as pending. The hour is not collected again, so a long run that fails afterwards does not change that hour's numbers.
  • Workflows are told apart by their workflow file (for example lint.yml), not by the run's display name.

Setting it up:

  • Token: use a GitHub token with read access to Actions on every repository you want to monitor.
  • Which repositories: the data source's default_owner is the organization or user, and repo_filter lists the repositories to monitor, comma separated. When repo_filter is empty, all non-archived repositories of the owner are monitored, up to 100. With more than 100, collection fails with a message asking you to set repo_filter.
  • API usage: each monitored repository costs at least one GitHub API call per hour, shared with the data source's other reads.
  • Unreadable repositories: a repository the token cannot read, or that has Actions disabled, is skipped and named in the collection job's message; the others are still collected. A rate limit or a GitHub server error fails the job instead, so that hour is collected again later.

GitHub Operational Issues​

A GitHub data source can also collect GitHub Operational Issues, a snapshot taken every hour of the open issues that carry an operational label, per repository. It shows two charts, one line per repository:

  • Open Operational Issues: how many open issues carry the label.
  • Oldest Open Age (days): how long the oldest of them has been open. Blank when none are open.

Details:

  • Label: set operational_label on the data source to choose the label. It defaults to operational.
  • Pull requests are not counted, even when they carry the label.
  • Snapshot: each point is the state at the time of collection, not a history of issues opened and closed in between.
  • Nothing open: a repository with no open labelled issues shows 0, so a flat zero means "none open", not "not collected".
  • Repositories, token and API usage work as for GitHub CI Runs above; this group needs read access to Issues instead of Actions.
  • A repository with more than 10,000 open issues carrying the label is skipped and named in the collection job's message, rather than being reported as 10,000.

GitHub Runners​

A GitHub data source can also collect GitHub Runners, a snapshot taken every hour of your organization's self-hosted runners. It is off by default: tick GitHub Runners under the Workload tab when you create the data source or edit it. A warning next to the checkbox names the token permission it needs. It shows two charts, one line per runner:

  • Self-hosted runners online (1 = online): 1 when the runner is connected to GitHub, 0 when it is offline.
  • Self-hosted runners busy (1 = running a job): 1 when the runner is running a job, 0 when it is idle.

Details:

  • Alerting: an alert on online below 1 fires when a runner drops off. A runner that has been removed from GitHub stops appearing at all, so it shows no rows rather than 0.
  • Snapshot: each point is the state at the time of collection, not a history of what happened in between.
  • Organization-wide: runners are read for the data source's default_owner, so it does not use repo_filter. The owner must be an organization; a personal account has no organization runners.
  • Ephemeral runners (a new runner name for every job, as with autoscaled runners) are not supported: each one would add a new line to the charts, and an online alert would never fire for them.
  • Limit: up to 1,000 runners are read. With more than that, collection fails with a message saying so, rather than reporting 1,000.

Token permission:

  • Use a fine-grained token with the organization permission Self-hosted runners: read. That is enough for this group.
  • A classic token needs the admin:org scope instead. It gives full organization admin, so prefer the fine-grained token.

Not monitored:

  • When the token lacks that permission, or the owner is not an organization, the group is not collected and the collection job still succeeds. The job's message reads GitHub Runners: not monitored, followed by the permission it needs. The charts stay empty until you grant it.
  • Because it is off by default, existing GitHub data sources are not affected. This message appears only on sources where you turned Runners on.
  • A rate limit or a GitHub server error fails the job instead, so that hour is collected again later.

Hetzner Cloud Workload​

A Hetzner Cloud data source can collect two workflows: Hetzner Cost Workload (daily), an estimate of cost per service type, and Hetzner Inventory Workload (hourly), which shows how many servers, volumes and load balancers the project has, how many servers are not running, and how many volumes are not attached to a server.

The inventory shows the latest hourly snapshot rather than a full history, and it leaves out Robot (dedicated) servers, floating IPs and networks. See Hetzner Cloud for the charts and what they leave out.

Cost Optimization​

Workload Analyzer provides insights into cost efficiency by identifying idle resources. If the analyzer indicates lower workload usage, responsible individuals in the organizations may choose to lower the resources of the data sources thereby reducing the operating costs yet maintaining reasonable performance.

User-Friendly Interface​

Workload Analyzer features an intuitive graphical interface (GUI) that simplifies the analysis process. Users can easily navigate through various metrics, query performance, query plan, storage usage, execution time, IO, etc.

MySQL Workload Graphs​

PostgreSQL Workload Graphs​

Use Cases​

Video Demonstration​