Cluster Monitoring

Monitor cluster usage and status down to each node

Overview

VESSL Clusters comes with a built-in cluster dashboard that provides a visualization of cluster usage and status down to each node and workload. This is enabled by the VESSL Cluster Agent which sends real-time information about the clusters and workloads running on the cluster such as node specifications and model metrics.

Dashboard setup is done automatically as you integrate your cloud or on-prem servers using vessl cluster create command.

Users on the Enterprise can use VESSL's custom cluster agent to route the monitoring information to your monitoring tools like Datadog and Grafana.

Contact us at [e-mail] or through our community Slack for more details.

Cluster-level Monitoring

Multi-cluster monitoring of resource usage and ongoing workloads is available under 🗂️ Clusters. Here, you can get an overview of the integrated clusters.

• Cluster status — Connection and incident status of a cluster.

• Available nodes — Available number of worker nodes.

• Real-time resource usage — Real-time resource usage of CPU cores, RAM, and GPUs.

• Ongoing workloads by type — The number of running notebook servers (Workspaces) and training jobs (Experiments).

Clicking the cluster guides you to the Summary tab which holds more detailed information about the cluster.

(1) Summary

The summary section presents the basic information about the cluster including the connection and incident status.

(2) Quotas & Usage

Quotas & Usage shows the organization-wide and personal resource quota for the cluster, including the number of GPU hours and occupiable GPUs and CPUs. This is set by the organization admin. Refer to our next section in the documentation VESSL Cluster's features on cluster administration.

(3) Resource Statistics

This section shows you how much CPU, GPU, and memory have been requested (and allocated) and are currently being used.

Note that when you are using VESSL Workspaces (notebook servers) you may be occupying a node without actively using the resources — you are only actively using the resources only when the cell is running.

(4) Workloads

This section shows all ongoing workloads on the cluster with information on the occupying node, resource consumption, creator, and the created date. If you are an organization admin, clicking the workload name guides you to the detailed workload page under 🗂️ Projects or 🗂️ Workspaces.

Node-level Monitoring

Under Nodes, you can view all the worker nodes tied to the cluster with their real-time CPU, Memory, and GPU usage, ongoing workloads by their type, and incident status. You can select the checkbox to get more in-depth information.

(1) Metadata

(2) System metrics

(3) Workloads

(4) Issues

Workload-level Monitoring

Under Workloads, you can view the workload log related to the cluster with the current status, occupying node, resource consumption, and a visualization of the usage history.

If you are on the Enterprise plan and wish to send the cluster information collected by VESSL Cluster Agent to your central infra monitoring tool such as Datadog and Grafana, contact us at [e-mail].