ONNX models (experimental)

This chapter is experimental. A trained model is baked into the code at compile time instead of being loaded at run time (no loading code, and no file to ship alongside it).

Two attributes

Attributes	Where it runs	Result
[Onnx("…")]	Udon	The result can be read back as numbers. Suited to small models
[OnnxGpu("…")]	GPU	Much larger models fit, but the result lands in a buffer, so using it as numbers takes one readback step

Pick the former when you read the values and branch on them, as in gesture recognition, and the latter when the result goes straight to the screen, as in reworking an image.

Generated names

Model file name	Prefix that is added
gesture.onnx	Gesture
hand_pose.onnx	HandPose

Underscores and the extension are dropped.

How to bake one in

1. Put the exported .onnx in the same folder as the script that will carry the attribute
2. Mark the class partial and save it with the attribute alone
3. Return to Unity: the import runs and the generated file appears next to the script
4. Write the code that uses the generated names

Writing both at once	Writing steps 2 and 4 in one go is a compile error, because the generated names do not exist yet
When it is rejected	No generated file is written, and the reason it was not accepted appears in the Console
When the model is swapped	Step 3 runs again on its own, and the element-count constants take their new values

The attribute that runs on Udon

The inputs and outputs are passed as arrays, and the result is read back as numbers.

The example model (gesture recognition)

Model file	gesture.onnx, in the same folder as Judge.cs
Input	pose — 12 elements (shape [1,12])
Output	scores — 4 elements (shape [1,4])
The generated file	Judge.Gesture.Model.cs

The script carrying the attribute (gesture recognition)

using Tsukimi;
using UnityEngine;

// The attribute argument is the path to the model file, seen from this script.
// Write two or more attributes and each model gets its own set of names.
[Onnx("gesture.onnx")]
public partial class Judge : TsukimiBehaviour
{
    // The generated file becomes the rest of this class, so partial is required.

    // Where the input values come from. Assign 12 of them in the Inspector.
    public Transform[] joints;

    // A value holding the inputs and outputs together. The generated file declares the type.
    private GestureIO io;

    // The chosen index. Other behaviours can read it.
    public int chosen;

    void Start()
    {
        io = new GestureIO();
        // The arrays are allocated once. After that the same arrays are overwritten.
        // Use the generated constants for the element counts (they change with the model).
        io.In0 = new float[SizeIn0];
        io.Out0 = new float[SizeOut0];
    }

    public void Decide()
    {
        // Fill every input. An element left unfilled is computed with its previous value.
        for (int i = 0; i < SizeIn0; i++) io.In0[i] = joints[i].localPosition.y;

        // Run it. Every layer advances before this returns.
        RunGesture(io);

        // The results land in io.Out0. Here the index of the largest component is picked.
        chosen = 0;
        for (int i = 1; i < SizeOut0; i++)
            if (io.Out0[i] > io.Out0[chosen]) chosen = i;
    }
}

The generated file (gesture recognition)

This is an excerpt, with the weights and the layer bodies left out.

// This file was generated from an ONNX model. Manual edits are lost on the next generation.
// Source model: gesture

using UnityEngine;

// The type holding the inputs and outputs. The original tensor names and shapes stay in comments.
public struct GestureIO
{
    public float[] In0;   // pose [1,12]
    public float[] Out0;   // scores [1,4]
}

public partial class Judge
{
    // The element counts used when allocating the arrays. Fixed at compile time.
    public const int SizeIn0 = 12;   // [1,12]
    public const int SizeOut0 = 4;   // [1,4]

    // It is private, so it cannot be called from outside without filling the inputs.
    private void RunGesture(GestureIO io)
    {
        float[] In0 = io.In0;   // pose
        float[] Out0 = io.Out0;   // scores

        // The trained weights appear as assignments into arrays.
        float[] c0 = new float[12];
        c0[0] = 0.0002085293f;
        c0[1] = 0.71990794f;
        // …(one for every weight)…

        // Arrays held between layers. Layers that pack four at a time become Vector4.
        float[] t5 = new float[12];   // centered [1,12]
        Vector4[] t7 = new Vector4[6];   // g0 [1,24]

        // The layers appear one at a time, labelled with their name and operator in the original model.
        // center: Elementwise  pose mean -> centered
        for (int i = 0; i < 12; i++)
        {
            t5[i] = In0[i] - c0[i];
        }

        // …(one for every layer)…

        // out: Copy  g2 -> scores
        for (int i = 0; i < 4; i++)
        {
            Out0[i] = t11[i];
        }
    }
}

The generated names (gesture recognition)

Generated names	Description
GestureIO	A type holding the inputs and outputs. The contents are arrays, named In0, Out0, and so on
SizeIn0 / SizeOut0	The element count of each array. The arrays are allocated once in Start
RunGesture(io)	Runs on the inputs in that value and writes the outputs back into the same value
In0 / In1 …	The number follows the order of the model’s inputs. The outputs follow the same rule

Where it can be called	The method that runs cannot be called from outside. This prevents running it without assigning the inputs
Cost	The computation advances one step at a time on Udon. Measure it with the profiler before running it every frame
Where the weights are kept	The weights are constants in the code. The model file does not have to ship with the world

The attribute that runs on the GPU

An image goes into a buffer, and the result comes back in another buffer.

The example model (an image filter)

Model file	filter.onnx, in the same folder as Sharpen.cs
Input	image — a 64×64 image with 3 components (shape [1,3,64,64])
Output	out — an image of the same size with 3 components (shape [1,3,64,64])
Layers	Four convolutions. The channel count in between grows 8 → 16 → 16
The generated file	Sharpen.Filter.Model.cs

The script carrying the attribute (an image filter)

using Tsukimi;
using UnityEngine;

// The attribute argument is the path to the model file, seen from this script.
[OnnxGpu("filter.onnx")]
public partial class Sharpen : TsukimiBehaviour
{
    // The source image and the surface the result is shown on. Assign them in the Inspector.
    public Texture source;
    public Renderer display;

    void Start()
    {
        // Creates the buffers. Their sizes come from the model, so you do not give them.
        LoadFilter();

        // Puts the source image into the input buffer. A different size is scaled to fit.
        Gpu.Load(FilterInput, source);
    }

    void Update()
    {
        // Runs one pass. One kernel per layer runs in order.
        RunFilter();

        // The result only stays in FilterOutput, so it is shown on the surface as it is.
        Gpu.Show(FilterOutput, display);
    }
}

The generated file (an image filter)

This is an excerpt, with the layer bodies left out.

using Tsukimi;
using UnityEngine;

public partial class Sharpen
{
    /// Input. Write values from 0 to 1 (mapped to 0 to 1).
    public GpuBuffer2D FilterInput;
    /// Output. Read values from 0 to 1 (corresponding to -2.39134 to 1.3269).
    public GpuBuffer2D FilterOutput;

    // Buffers between layers. The names begin with the model's name.
    private GpuBuffer2D Filterb1;
    // …(one for every layer)…

    // The weight table for each layer.
    public Vector4[] Filterw0;
    // …(one for every layer)…

    /// Loads the weights and creates the buffers. Call once from <c>Start</c>.
    private void LoadFilter()
    {
        // The entry is 64×64. The channel count is 4 or fewer, so it does not extend sideways.
        FilterInput = Gpu.Buffer(64, 64);
        // The next layer has 8 channels — two bands of 4 components — so the width doubles.
        Filterb1 = Gpu.Buffer(128, 64);
        // …(one for every layer)…
        FilterOutput = Gpu.Buffer(64, 64);
    }

    // One kernel is generated per layer.
    [Kernel]
    static Color4 FilterLayer0(KernelId id, GpuBuffer2D prev, [Capacity(128)] Vector4[] t)
    {
        // …(the multiply-accumulates of the convolution follow)…
    }

    /// Runs the model once. Write <c>FilterInput</c> before calling, then read <c>FilterOutput</c>.
    private void RunFilter()
    {
        Gpu.Run(nameof(FilterLayer0), Filterb1, FilterInput, Filterw0);
        // …(one for every layer)…
    }
}

The generated names (an image filter)

Generated names	Description
FilterInput	The input buffer (the same type as in Running a GPU kernel). The range of values to assign to a cell is written in the generated file, next to the input field.
FilterOutput	The output buffer. Read its cells as values from 0 to 1
LoadFilter()	Creates the buffers and loads the weights. Call it once from Start
RunFilter()	Runs one pass
FilterLayer0	The kernel for each layer. Gpu.Run launches it, so you never call it directly
FilterStyles	Generated only when 2 or more paths are listed. The number of paths listed
UseFilter(int style)	Generated only when 2 or more paths are listed. Chooses which weights to use by their position in the list (counting from 0). Right after LoadFilter() it is 0

Value range	For most models, assigning a value between 0 and 1 to a cell passes it through correctly, because the 8-bit steps line up exactly with the range the model accepts. A model that takes its input as 16-bit floats also passes values outside 0 to 1 through unchanged. Which one applies is decided by the model, and is written in the input field of the generated file.
What the output means	The range the output values actually stand for is written in the generated file.
Reading it back	The result lands in a buffer. To use it as numbers, take it out with Gpu.Texture and hand it to the runtime’s asynchronous readback
Swapping weights	Models that share one graph and differ only in their weights become one when their paths are listed, as in [OnnxGpu("a.onnx", "b.onnx")]. The buffers and kernels stay as one set, and only the weights are kept once per listed model

Supported operators

A model containing an operator not listed here is a compile-time error.

Operators for the Udon attribute

Operators	Condition
Gemm	A and B are 2-D. transA, transB, alpha and beta are handled
MatMul	A 1-D operand is not accepted. Leading axes are broadcast
Add	The two inputs are broadcast
Sub	The two inputs are broadcast
Mul	The two inputs are broadcast
Relu	No conditions
Identity	No conditions
Reshape	The shape must be a constant (a shape passed at run time is not accepted)
Flatten	axis must be within the input’s rank

Operators for the GPU attribute

Operators	Condition
Conv	kernel_shape has equal height and width. strides and dilations are handled. group must be 1, or equal to both the input and output channel counts (depthwise). auto_pad is not accepted
DepthToSpace	Only the DCR mode. The block size is a power of 2, and the channel count must be divisible by its square
Concat	Only along the channel axis. Up to 4 inputs, each with a channel count that is a multiple of 4
Add	Both inputs come from earlier operators and have the same shape. Adding a constant is not accepted
QuantizeLinear	Read as the quantization scale, not as a computation. Scale and zero point are constants
DequantizeLinear	Read as the quantization scale, not as a computation. Putting a Cast (float16) and Cast (float32) pair in place of quantization carries that edge as half-precision floats
Pad	Only reflect, placed directly before a Conv with no padding of its own (it is folded into the Conv)
InstanceNormalization	scale and B are constants
Resize	Only linear interpolation. The coordinate_transformation_mode values half_pixel, pytorch_half_pixel, align_corners and asymmetric are handled. Only height and width can change
PRelu	The slope is a constant with either 1 value or one per channel
MaxPool	kernel_shape has equal height and width, and dilations are not accepted. The positions of the maxima (Indices) cannot be output
Elu	8-bit input only
Relu	Half-precision float input is accepted as well
LeakyRelu	Half-precision float input is accepted as well
Tanh	8-bit input only
Sigmoid	8-bit input only
Clip	The bounds are constants. Half-precision float input is accepted as well

Models accepted on the GPU

Forms that produce an error	What to do	Reason
Not quantized	Quantize the weights to 8 bits, and quantize the output to 8 bits as well, before exporting	Weights are baked in as 8-bit integers. The input and the values between layers can be carried as 8 bits or as half-precision floats (a Cast pair), but the output is received as 8 bits
The scale is decided at run time	Export it in a form that is fixed at export time	To bake it in, the value steps have to be fixed at compile time
Not an 8-bit integer	Quantize to 8-bit integers	Even at 8 bits, a format that represents fractions is not accepted
The shape is dynamic	Fix the input shape before exporting	When an axis length is decided at run time, the texture size cannot be determined
A listed model differs from the first in more than its weights	Export it with the same graph, changing only the weights	Names the layer that differs. A model whose scales built into the computation differ is also an error here
An unsupported operator is present	Rewrite it into a form that does not use that operator	It names what was present. It never leaves one out and passes the model anyway

The error speaks in terms of the model, so fix the exporting side.

Limits when running on the GPU

One cell holds 4 components	For a model whose channel count exceeds 4, the texture extends sideways
Where the weights are kept	Small layers are kept as constants and large ones as textures. Which one applies is decided per layer
Stride	strides other than 1 are handled for Conv and MaxPool
Width	For a layer with more than 4 channels it must be a power of 2. Concat and DepthToSpace need it regardless of the channel count. Height has no restriction
Class declaration	partial is required. The generated file becomes the rest of that class

The generated file

Name	The script name joined with the model name. Writing the attribute for gesture.onnx in Judge.cs gives Judge.Gesture.Model.cs
Where it is placed	The same folder as the script carrying the attribute
Where the model is looked for	The path in the attribute is looked for next to the script first, then in the project folder
Regeneration	It is regenerated when either the script or the model changes, and removed when the attribute is deleted. Do not edit it by hand.
Size	The weights become code as they are, so it runs to more lines than the number of weights, and all of it is read on every compile
Order of writing	Writing the attribute and the code that uses the generated names at the same time is a compile error. Save the attribute on its own first.

Accuracy

The result is not guaranteed to match the original model. Values are rounded when quantized to 8 bits, and values carried in half precision follow the GPU’s own rounding when stored, so they differ from what the exporting runtime computes by that much.