The mission of OVL is to provide a framework for easily defining high-performance, python-native operators that can work transparently with TensorFlow on a variety of different processor architectures. OVL operators are written by users in OVL’s python-embedded domain specific language (DSL). The programming model of operators is similar to that of CUDA or OpenCL - conceptually an operator is a stateless function that is mapped over a set of user-defined thread or worker indices. Operators are defined as the python interpreter encounters them but are evaluated lazily, allowing for numerical and performance optimizations to be applied to the entire graph of defined operators before evaluation time.
The OVL DSL is designed to strike a balance between productivity, performance, and portability and as such there are some important restrictions that differentiate it from similar approaches:
- The DSL is strongly typed and tensor shapes are part of the type system. This means that inputs and outputs must have a concrete shape - there is no support for dynamic tensor sizing.
- Individual worker elements cannot communicate and, as such, there are no synchronization barriers.
- Operators have no persistent state between calls.
- System environment information
- Library versions
pip show tensorflow opveclib
- Logs, e.g. by using
logging.basicConfig(filename='output.log',level=logging.DEBUG)in your code
- Instructions or code snippets for reproducing the issue
OVL currently uses the python interpreter to parse operators into its own intermediate representation, an approach which comes with some limitations. These limitations result in a programming API that has two common pitfalls that raise a syntax error when encountered. These pitfalls are highlighted here to prevent confusion, but examples of proper use cases are detailed in the examples section below. These limitations may eventually disappear (i.e. not throw a syntax error and work as expected) as improvements are made to library internals.
Once variables are defined within the operator, the python-native assignment operator `` = `` cannot be used on them. Use the `` <<= `` operator instead. For example, the following should raise an error:
import opveclib as ovl x = ovl.variable(ovl.float32) x = 2.0
The correct way to assign a value to a previously-defined variable is as follows:
import opveclib as ovl x = ovl.variable(ovl.float32) x <<= 2.0
else statements cannot be used with OVL expressions. OVL
conditionals must be invoked from within a
with statement. More specifically, the following should raise an error:
import opveclib as ovl x = ovl.variable(ovl.float32) if x < -1: x <<= -1 elif x > 1: x <<= 1
The correct way to use conditionals is as follows:
import opveclib as ovl x = ovl.variable(ovl.float32) with ovl.if_(x < -1): x <<= -1 with ovl.elif_(x > 1): x <<= 1
Guide by examples¶
Designing the simplest OVL operator requires understanding a few key concepts and their corresponding implementations in the OVL API. This example takes the absolute value of an input tensor and shows all of the basic functionality necessary to create and use an operator. New concepts introduced here include:
- An OVL operator is defined by creating a python function and decorating it with the
- Arguments to the operator are the tensors that it will operate on at evaluation time.
- Output tensors are the only thing that can be returned from operators. They are defined with the
- Operators are implicitly mapped over a set of workgroup positions. The workgroup shape must be statically
defined based on the arguments to the operator and must be either a single int or a list of ints. The
position_infunction is used to define the workgroup shape and returns a
PositionTensorobject which is used to identify the position of the current worker element. A workgroup shape can be defined to be any number of dimensions that makes sense for the problem at hand. A PositionTensor can be indexed to obtain the current worker’s position along a specific axis.
- Data elements in input tensors are accessed by indexing into them. Indexing into an input tensor yields a
Scalarcan be transformed into a new
Scalarby applying scalar operators to them, in this case the
- Output tensors are written to by setting
Scalarvalues at a specific position, in this case the position is just the
PositionTensor, but more complicated write patterns follow in other examples.
- Operators can be tested independently of the TensorFlow runtime using the OVL test infrastructure via the
evaluatefunction. Testing operators using this infrastructure is recommended since it isolates the operator from the TF runtime. An explicit
evaluatefunction is used so that operators can be lazily evaluated, increasing the opportunity for optimization.
- Operators are linked into the TensorFlow runtime by explicitly converting operator outputs to TensorFlow tensors
import numpy as np import tensorflow as tf import opveclib as ovl @ovl.operator() def absolute(input_tensor): # define the output tensor output_tensor = ovl.output_like(input_tensor) # define the workgroup shape and get workgroup position reference wg_position = ovl.position_in(input_tensor.shape) # read input element at current workgroup position input_element = input_tensor[wg_position] # apply absolute function abs = ovl.absolute(input_element) # set the output element output_tensor[wg_position] = abs return output_tensor # define a numpy input in_np = np.arange(-3, 4, dtype=np.float32) # apply the operator out = absolute(in_np) # lazily evaluate with the OVL test infrastructure print(ovl.evaluate([out])) # explicitly convert to tensorflow tensor out_tf = ovl.as_tensorflow(out) # lazily evaluate using tensorflow sess = tf.Session() print(sess.run([out_tf]))
[ 3. 2. 1. 0. 1. 2. 3.] [ 3. 2. 1. 0. 1. 2. 3.]
Many operators depend on an input value that does not change throughout the lifetime of the operator, and is not
dependent on the data values of the input tensors.
Examples include summing along a constant axis, applying a constant threshold,
and applying a constant power to an input tensor. This example implements the
power function which raises an
input tensor to a specified power and introduces the new concept of a constant:
- Constant arguments are differentiated from input tensors by explicitly giving the operator function argument a default
value. If the default value is
Nonean error will be raised if the argument is not set when the operator is applied. When calling the operator, constants are specified by passing them to the operator as keyword arguments.
import numpy as np import tensorflow as tf import opveclib as ovl @ovl.operator() def power(input_tensor, exponent=None): # define the output tensor output_tensor = ovl.output_like(input_tensor) # define the workgroup shape and get workgroup position reference wg_position = ovl.position_in(input_tensor.shape) # read input element at current workgroup position input_element = input_tensor[wg_position] # apply power function and set the output element output_tensor[wg_position] = ovl.power(input_element, exponent) return output_tensor # define a numpy input in_np = np.arange(-3, 4, dtype=np.float32) # apply the operator # Note that constants must be explicitly set as keyword arguments out = power(in_np, exponent=2) # lazily evaluate with the OVL test infrastructure print(ovl.evaluate([out])) # explicitly convert to tensorflow tensor out_tf = ovl.as_tensorflow(out) # lazily evaluate using tensorflow sess = tf.Session() print(sess.run([out_tf]))
[ 9. 4. 1. 0. 1. 4. 9.] [ 9. 4. 1. 0. 1. 4. 9.]
Conditionals and Variables¶
Operators can exhibit control flow and use thread-local memory by using OVL conditionals and variables. This example
clip function which clips the values of the input tensor to within the specified boundaries.
The following new concepts are illustrated here:
Variable, which is a worker-local scalar that has state which can be set with the
- The conditionals
with else_()which are used to conditionally execute a segment of the operator. Python native conditionals can be used from within an operator, but only when they are operating on a constant. This example differentiates between the two types of conditional.
forbid_none_valued_constantsoption to the
operatordecorator which overrides the default behavior to and allows
Noneas a valid constant value.
import numpy as np import opveclib as ovl # override default behavior and allow None valued constants @ovl.operator(forbid_none_valued_constants=False) def clip(arg, threshold1=None, threshold2=None): pos = ovl.position_in(arg.shape) clipped = ovl.output_like(arg) # define a worker local variable clipped_val = ovl.variable(0, arg.dtype) # assign the value of variables with the the <<= operator clipped_val <<= arg[pos] # python if statements can be used on constants to control how the operator behaves if threshold1 is not None: # when using conditionals on OVL expressions, you must use the OVL "with if_" function # these conditionals are evaluated at run time and are input-data dependent with ovl.if_(clipped_val < threshold1): clipped_val <<= threshold1 # this python native if statement is used to control how the operator behaves at definition time if threshold2 is not None: # this conditional controls how the operator deals with run time data with ovl.if_(clipped_val > threshold2): clipped_val <<= threshold2 clipped[pos] = clipped_val return clipped # define a numpy input in_np = np.arange(-3, 4, dtype=np.float32) # apply the operator # not all constants have to be supplied here since None value constants are permitted out1 = clip(in_np, threshold1=-1) out2 = clip(in_np, threshold2=1) out3 = clip(in_np, threshold1=-1, threshold2=1) # lazily evaluate with the OVL test infrastructure res1, res2, res3 = ovl.evaluate([out1, out2, out3]) print(res1) print(res2) print(res3)
[-1. -1. -1. 0. 1. 2. 3.] [-3. -2. -1. 0. 1. 1. 1.] [-1. -1. -1. 0. 1. 1. 1.]
Loops and non-local IO¶
OVL supports iterating over ranges and defining arbitrary workgroup shapes that may or may not be the same size as the input or output tensors. This simple example of a naive implementation of matrix-vector multiplication show the use of two new concenpts:
- The arange() iterator which works like the python native range but allows for runtime iteration based off of either constants or data from one of the input tensors
- Arbritrary workgroup shapes - in this case the workgroup shape is determined by the number of rows in the matrix.
- Non-local IO is possible, in the sense that in this example each worker reads in multiple elements from the input matrix and vector.
import numpy as np import opveclib as ovl @ovl.operator() def matmul(mat, vec): # assert type properties of the two input tensors assert mat.rank == 2 assert vec.rank == 1 assert mat.shape == vec.shape assert mat.dtype == vec.dtype rows = mat.shape cols = mat.shape # define the output to be one dimensional y = ovl.output(rows, mat.dtype) # define a number of workers equal to the number of rows row = ovl.position_in(rows) # define the accumulator and iterate over the matrix columns accum = ovl.variable(0, mat.dtype) for col in ovl.arange(cols): # accumulated the product of each element accum <<= accum + mat[row, col]*vec[col] y[row] = accum return y m = np.array([[0, 1, 2], [3, 4, 5], [6, 7, 8]], dtype=np.float32) x = np.array([1, 2, 3], dtype=np.float32) out = matmul(m, x) print(ovl.evaluate([out]))
[ 8. 26. 44.]
Similar to the OVL Variable, OVL also supports dynamically declared worker-local tensors of any dimension that use thread-local memory. Worker local tensors are generally small (on the order of hundreds of bytes) and should fit in register space on device or L1 cache.
An OVL worker local tensor is defined by its shape and type and can be initialized to either all zeros or all ones using the zeros() or ones() functions.
import opveclib as ovl # declare a 1x3 worker local tensor of type float32, initialized to all zeros a = ovl.zeros((1, 3), ovl.float32) for col in ovl.arange(3): a[col] = 0.9
An OVL gradient operator is a special type of operator. It is defined by creating a python gradient function and
decorating it with the
operator() decorator as well as the
gradient() decorator. The arguement to the
gradient decorator function is the name of the operator it is the gradient for. The gradient function
must compute the gradients with respect to each of the operators’ inputs, given gradients with respect to the operators’
outputs. Its arguments are the operator input tensors as well as the output gradient tensors from the next higher
layer in the case of back propagation through a neural network.
Below is a simple operator that computes the function
and its gradient, and tests the gradient against tensorflow
import numpy as np import opveclib as ovl import tensorflow as tf @ovl.operator() def func(x, y): assert(x.shape == y.shape) pos = ovl.position_in(x.shape) z = ovl.output_like(x) # compute the function z[pos] = x[pos] * x[pos] + 3.0 * y[pos] return z @ovl.gradient(func) @ovl.operator() def func_grad(x, y, dz): assert(x.shape == y.shape) assert(x.shape == dz.shape) pos = ovl.position_in(x.shape) dx = ovl.output_like(x) dy = ovl.output_like(y) # compute the gradients w.r.t each input dx[pos] = 2.0 * x[pos] * dz[pos] dy[pos] = 3.0 * dz[pos] return dx, dy # define some input values a = tf.constant(np.arange(5.0)) b = tf.constant(np.arange(5.0)) grad_above = tf.constant(np.random.random(5)) # run the operator in tensorflow with tf.Session() as s: func_ovl = ovl.as_tensorflow(func(a, b)) # define a tensorflow reference operation func_tf = a * a + 3.0 * b # register the gradients of the ovl operator and the refenence in tensorflow func_grad_ovl = tf.gradients(func_ovl, [a, b], grad_ys=grad_above) func_grad_tf = tf.gradients(func_tf, [a, b], grad_ys=grad_above) # evaluate the outputs func_grad_ovl, func_grad_tf = s.run([func_grad_ovl, func_grad_tf]) assert np.all(np.equal(func_grad_ovl, func_grad_tf))
Typical modern GPUs are able to perform arithmetic operations faster than transfers of a data value in or out of host memory, and are thus memory bandwidth limited. If 2 operators that are performed in sequence can be merged into a single operator that shares data values in device memory, performance can be improved. In an application using multiple OVL operators in sequence, the OVL optimizer will automatically merge any operators that it can. 2 OVL operators in sequence, from and to, will be merged if:
- the workgroup shape of both operators is the same
- the write indexing pattern of the from operator matches the read indexing pattern of the to operator
- the tensor shape of the output of the from operator matches the tensor shape of the input to the to operator
Optimization by automatically merging OVL operators is determined by the opt_level flag that is passed to the
as_tensorflow functions. Merging is performed when opt_level >= 3. This is the default optimization level.
When operators are merged, their gradient operators will also be merged when possible.