YAML Schema Reference

YAML Schema reference for serving

Revision YAML Field Types

Message

Write a message for the Serving Revision. We recommend writing an identical message for each revision to distinguish them.

Name	Type	Required	Description
message	str	Requried	Description of the revision.

message: vessl-serve-using-yaml

Launch_immediately

Determines whether the revision will be deployed immediately.

Name	Type	Required	Description
launch_immediately	boolean	Requried	True if revision is launch immediately.

launch_immediately: true

Image

The name of the docker image that will be used for inference. You can also use a custom docker image.

Name	Type	Required	Description
image	string	Requried	Docker image url.

image: quay.io/vessl-ai/ngc-pytorch-kernel:22.10-py3-202306140422

Resources

Write down the compute resources you want to use for Serving. You can specify the resources you want to use in the Cluster settings.

Name	Type	Required	Description
cluster	string	Optional	The cluster to be used for the run. (default: VESSL-managed cluster)
name	string	Optional	The resource spec name that specified in VESSL. If the name is not specified, we will offer the best option for you based on cpu, memory, and accelerators.
cpu	string	Optional	The number of cpu cores.
memory	string	Optional	The memory size in GB.
accelerators	string	Optional	The type and quanity of the GPU to be used for the run.
spot	boolean	Optional	Whether to use spot instances for the run or not.

resources:
  cluster: vessl-tmap-gi-aiml-stg
  accelerators: T4:1 # using T4 with 1 GPU
  spot: true

💡 You can list available clusters or resource specs with the CLI command: `vessl cluster list` or `vessl resource list`.

Volumes

Write the datasets and volumes mounted in the Revision container when the Revision is deployed.

Prefix	Type	Required	Description
git://	string	Optional	Mount a git repository into your container. The repository will be cloned into the specified mount path when container starts.
vessl-dataset://	string	Optional	Mount a dataset stored in VESSL. Replace {organizationName} with the name of your organization and {datasetName} with the name of the dataset.
s3://	string	Optional	Mount an AWS S3 bucket into your container. Replace {bucketName} with the name of your S3 bucket and {path} with the path to te file or folder you want to mount.
local://	string	Optional	Mount a file or directory from the machine where you are running the command. This can be useful for using configuration files or other data that is not in your Docker image.
hostpath://	string	Optional	Mount a file or directory from the host node’s filesystem into your container. Replace {path} with the path to the file or folder you want to mount.
nfs://	string	Optional	Mount a Network File System(NFS) into your container. Replace {ip} with the IP address of your NFS server and {path} with the path to the file or folder you want to mount.
cifs://	string	Optional	Mount a Command Internet File System(CIFS) into your contianer. Replace {ip} with the IP address of your NFS server and {path} with the path to the file or folder you want to mount.

volumes:
	/root/git-examples: git://github.com/vessl-ai/examples
	/input/data1: hostpath:///opt/data1
	/input/config: local://config.yml
	/input/data2: nfs://192.168.10.2:~/
	/input/data3: vessl-dataset://{organization_name}/{dataset_name}
	/output:
	  artifact: true

You can also add an artifact flag to indicate whether the directory /output should be treated as an output artifact. Typically, volumes store model checkpoints or key metrics.

Run

Write down what commands you want to run on the service container when the Revision is deployed.

Name	Type	Required	Description
workdir	string	Optional	The working directory for the command.
command	string	Required	The command to be run.

run:
  - workdir: /root/git-examples
    command: |
      python train.py --learning_rate=$learning_rate --batch_size=$batch_size

Env

Write down the environment variables that will be set in the Revision Service container.

Name	Type	Required	Description
env	map	Optional	Key-value pairs for environment variables in the run container.

env:
  learning_rate: 0.001
  batch_size: 64
  optimizer: sgd

Ports

Write down the ports and protocols that the Revision Service container should open.

Name	Type	Required	Description
name	string	Required	The name for the opening port.
type	string	Required	The protocol the port will use.
port	int	Required	The number of the port.

ports:
  - name: web-service
    type: http
    port: 8000
  - name: web-service-2
    type: http
    port: 8001
...

Autoscaling

Sets the value for how the Revision Pod will autoscale.

Name	Type	Required	Description
min	string	Required	Minimum number of Pods to autoscale.
max	string	Required	Maximum number of Pods to autoscale.
metric	int	Required	Determine what conditions you want to autoscale under. You can select cpu, gpu, memory, and custom
target	int	Required	A metric threshold percentage. If the metric is above the target, then the Autoscaler automatically scale-out.

autoscaling:
  min: 1
  max: 3
  metric: cpu
  target: 50

Simple YAML example for revision

message: vessl-yaml-serve-test
launch_immediately: true

image: quay.io/vessl-ai/kernels:py39

resources:
  accelerators: T4:1
  spot: true

volumes:
  /root/examples:
    git:
      clone: https://github.com/vessl-ai/examples
      revision: 33a49398fc6f87265ac490b1cf587912b337741a

run:
  - workdir: /code/examples
    command: |
      python3 mnist.py

env:
  - key: TEST_ENV
    value: test

ports:
  - name: http
    type: http
    port: 8000

autoscaling:
  min: 1
  max: 1
  metric: cpu
  target: 50

Gateway YAML Field Types

Enabled

Name	Type	Required	Description
enabled	boolean	Required	Whether gateway is enabled or not.

enabled: true

Targets

Name	Type	Required	Description
number	string	Required	The revision number that the Gateway will use for routing.
port	string	Required	The port number that the gateway will use for routing.
weight	int	Required	The weight to determine how much traffic should be distributed.

targets:
  - number: 1
    port: 8000
    weight: 50
  - number: 2
    port: 8001
    weight: 50

Sample Gateway YAML Schema

enabled: true
targets:
  - number: 1
    port: 8000
    weight: 10
  - number: 2
    port: 8000
    weight: 90

Serving example with YAML

MNIST model mount example

message: Example serving from YAML
image: quay.io/vessl-ai/kernels:py310-202301160626
resources:
  name: cpu-m6i-large
volumes:
  /root:
    model:
      repo: vessl-mnist-example
      version: 2
run: vessl model serve vessl-mnist-example 2 --install-reqs --remote
env:
  - key: VESSL_LOG
    value: DEBUG
autoscaling:
  min: 1
  max: 3
  metric: cpu
  target: 60
ports:
  - port: 8000
    name: service
    type: http