Name Strings

    cl_ext_device_fission

Contributors

    Greg Bellows, IBM
    Ben Gaster, AMD
    Aaftab Munshi, Apple
    Ian Ollmann, Apple
    Ofer Rosenberg, Intel
    Brian Watt, IBM

Contact

    Ian Ollmann, iano 'at' apple 'dot' com

IP Status

    No known IP issues

Version

    Version 1, June 9, 2010

Number

    OpenCL Extension #11

Status

    Approved by all contributors

Extension Type

    OpenCL device extension

Dependencies

    None

Overview

    This extension provides an interface for sub-dividing an OpenCL device
    into multiple sub-devices.  There are a number of cases in which a typical 
    user would like to subdivide a device:
    
    1. To reserve part of the device for use for high priority / 
       latency-sensitive tasks 
    2. To more directly control the assignment of work to individual compute 
       units
    3. To subdivide compute devices along some shared hardware feature like a 
       cache

    Typically these are areas where some level of additional control is required
    to get optimal performance beyond that provided by standard OpenCL 1.1 APIs.
    Proper use of this interface assumes some detailed knowledge of the devices 
    in question.

Header File

    Extension interfaces and constants defined in cl_ext.h

New Procedures and Functions
    cl_int clReleaseDeviceEXT( cl_device_id device );

    cl_int clRetainDeviceEXT( cl_device_id device );

    cl_int clCreateSubDevicesEXT( 
        cl_device_id in_device,
        const cl_device_partition_property_ext * properties,
        cl_uint num_entries,
        cl_device_id *out_devices,
        cl_uint *num_devices );

New Types

    The following new types have been added for declaring device
    partitionining properties.

    typedef cl_bitfield cl_device_partition_property_ext;
    
New Tokens

    Accepted as a property name in the <properties> parameter of
    clCreateSubDeviceEXT:

        CL_DEVICE_PARTITION_EQUALLY_EXT                 0x4050
        CL_DEVICE_PARTITION_BY_COUNTS_EXT               0x4051
        CL_DEVICE_PARTITION_BY_NAMES_EXT                0x4052
        CL_DEVICE_PARTITION_BY_AFFINITY_DOMAIN_EXT      0x4053
    
    Accepted as a property name, when accompanying the
    CL_DEVICE_PARITION_BY_AFFINITY_DOMAIN_EXT property, in the <properties> 
    parameter of clCreateSubDeviceEXT:

        CL_AFFINITY_DOMAIN_L1_CACHE_EXT                 0x1
        CL_AFFINITY_DOMAIN_L2_CACHE_EXT                 0x2
        CL_AFFINITY_DOMAIN_L3_CACHE_EXT                 0x3
        CL_AFFINITY_DOMAIN_L4_CACHE_EXT                 0x4
        CL_AFFINITY_DOMAIN_NUMA_EXT                     0x10
        CL_AFFINITY_DOMAIN_NEXT_FISSIONABLE_EXT         0x100

    Accepted as a property being queried in the <param_name> argument of
    clGetDeviceInfo:

        CL_DEVICE_PARENT_DEVICE_EXT                     0x4054
        CL_DEVICE_PARITION_TYPES_EXT                    0x4055
        CL_DEVICE_AFFINITY_DOMAINS_EXT                  0x4056
        CL_DEVICE_REFERENCE_COUNT_EXT                   0x4057
        CL_DEVICE_PARTITION_STYLE_EXT                   0x4058

    Accepted as the property list terminator in the <properties> parameter of
    clCreateSubDeviceEXT:

        CL_PROPERTIES_LIST_END_EXT                      0x0

    Accepted as the partition counts list terminator in the <properties>
    parameter of clCreateSubDeviceEXT:

        CL_PARTITION_BY_COUNTS_LIST_END_EXT             0x0

    Accepted as the partition names list terminator in the <properties>
    parameter of clCreateSubDeviceEXT:

        CL_PARTITION_BY_NAMES_LIST_END_EXT              -1

    Returned by clCreateSubDevicesEXT when the indicated partition scheme is
    supported by the implementation, but the implementation can not further
    partition the device in this way.

        CL_DEVICE_PARTITION_FAILED_EXT                  -1057

    Returned by clCreateSubDevicesEXT when the total number of compute units
    requested exceeds CL_DEVICE_MAX_COMPUTE_UNITS, or the number of compute
    units for any one sub-device is less than 1.

        CL_INVALID_PARTITION_COUNT_EXT                  -1058

    Returned by clCreateSubDevicesEXT when a compute unit name appearing in a
    name list following CL_DEVICE_PARTITION_BY_NAMES_EXT is not in range.

        CL_INVALID_PARTITION_NAME_EXT                   -1059


Additions to Section 4.2 (Querying Devices)

       cl_int clReleaseDeviceEXT( cl_device_id device );

    clReleaseDeviceEXT decrements the <device> reference count. After the 
    reference count reaches zero, the object shall be destroyed and associated 
    resources released for reuse by the system. 
    
    clReleaseDeviceEXT returns CL_SUCCESS if the function is executed 
    successfully or the device is a root level device. It returns 
    CL_INVALID_DEVICE if the <device> is not a valid device.

        cl_int clRetainDeviceEXT( cl_device_id device );

    clRetainDeviceEXT increments the <device> reference count. 
    
    clReleaseDeviceEXT returns CL_SUCCESS if the function is executed 
    successfully or the device is a root level device. It returns 
    CL_INVALID_DEVICE if the <device> is not a valid device.

    CAUTION: Since root level devices are generally returned by a clGet call 
    (clGetDeviceIDs) and not a clCreate call, the user generally does not own a 
    reference count for root level devices. The reference count attached to a 
    device retured from clGetDeviceIDs is owned by the implementation. 
    Developers need to be careful when releasing cl_device_ids to always balance
    clCreateSubDevicesEXT or clRetainDeviceEXT with each call to 
    clReleaseDeviceEXT for the device. By convention, software layers that own 
    a reference count should be themselves responsible for releasing it.

        typedef cl_bitfield cl_device_partition_property_ext;
        cl_int clCreateSubDevicesEXT( 
                            cl_device_id in_device,
                            const cl_device_partition_property_ext * properties,
                            cl_uint num_entries,
                            cl_device_id *out_devices,
                            cl_uint *num_devices );

    clCreateSubDevicesEXT creates an array of sub-devices that each reference a 
    nonintersecting set of compute units within <in_device>, according to a 
    partition scheme given by the <cl_device_partition_property_ext> list. The 
    output sub-devices may be used in every way that the root device can be 
    used, including building programs, further calls to clCreateSubDevicesEXT 
    and creating command queues. They may also be used within any context 
    created using the in_device or parent/ancestor thereof. When a command 
    queue is created against a sub-device, the commands enqueued on that queue 
    are executed only on the sub-device.
    
    in_device - The device to be partitioned 

    num_entries - The number of cl_device_ids that will fit in the array pointed
        to by <out_devices>. If <out_devices> is not NULL, <num_entries> must be
        greater than zero.  
    
    out_devices - On output, the array pointed to by <out_devices> will contain 
        up to <num_entries> sub-devices. If the <out_devices> argument is NULL, 
        it is ignored. The number of cl_device_ids returned is the minimum of 
        <num_entries> and the number of devices created by the partition scheme.
    
    num_devices - On output, the number of devices that the <in_device> may be 
        partitioned in to according to the partitioning scheme given by 
        <properties>. If num_devices is NULL, it is ignored.

    properties - A zero terminated list of device fission {property-value, 
        cl_int[]} pairs that describe how to partition the device into 
        sub-devices.  <properties> may not be NULL. Only one of 
        CL_DEVICE_PARTITION_EQUALLY_EXT, CL_DEVICE_PARTITION_BY_COUNTS_EXT, 
        CL_DEVICE_PARTITION_BY_NAMES_EXT or 
        CL_DEVICE_PARTITION_BY_AFFINITY_DOMAIN_EXT may be used in the same 
        properties list. Available properties are:

        CL_DEVICE_PARTITION_EQUALLY_EXT - Split the aggregate device into as 
            many smaller aggregate devices as can be created, each containing N
            compute units. The value N is passed as the value accompanying this 
            property.  If N does not divide evenly into 
            CL_DEVICE_MAX_COMPUTE_UNITS then the remaining compute units are 
            not used.

            Example: To divide a device containing 16 compute units into two 
            sub-devices, each containing 8 compute units, pass:

                { CL_DEVICE_PARTITION_EQUALLY_EXT, 8, 
                  CL_PROPERTIES_LIST_END_EXT }
    
        CL_DEVICE_PARTITION_BY COUNTS_EXT - This property is followed by a 
            CL_PARTITION_BY_COUNTS_LIST_END_EXT terminated list of compute unit 
            counts. For each non-zero count M in the list, a sub-device is 
            created with M compute units in it.  
            CL_PARTITION_BY_COUNTS_LIST_END_EXT is defined to be 0.

            Example: to split a four compute unit device into two sub-devices, 
            each containing two compute units, pass:

                { CL_DEVICE_PARTITION_BY_COUNTS_EXT, 
                  2, 2, CL_PARTITION_BY_COUNTS_LIST_END_EXT,
                  CL_PROPERTIES_LIST_END_EXT }

            The first 2 means put two compute units in the first sub-device. The
            second 2 means put two compute units in the second sub-device. 
            CL_PARTITION_BY_COUNTS_LIST_END_EXT terminates the list of 
            sub-devices.  CL_PROPERTIES_LIST_END_EXT terminates the list of 
            properties.  The total number of compute units specified may not 
            exceed the number of compute units in the device.

        CL_DEVICE_PARTITION_BY NAMES_EXT - This property is followed by a list 
            of compute unit names. Each list starts with a 
            CL_PARTITION_BY_NAMES_LIST_END_EXT terminated list of compute unit 
            names.  Compute unit names are integers that count up from zero to 
            the number of compute units less one. 
            CL_PARTITION_BY_NAMES_LIST_END_EXT is defined to be -1. Only 
            one sub-device may be created at a time with this selector. An 
            individual compute unit name may not appear more than once in the 
            sub-device description.

            Example: To create a three compute unit sub-device using compute 
            units, { 0, 1, 3 }, pass:
    
                { CL_DEVICE_PARTITION_BY NAMES_EXT,
                  0, 1, 3, CL_PARTITION_BY_NAMES_LIST_END_EXT,
                  CL_PROPERTIES_LIST_END_EXT }

            The meaning of these numbers are, in order:
              0 the name of the first compute unit in the sub-device
              1 the name of the second compute unit in the sub-device
              3 the name of the third compute unit in the sub-device
              CL_PROPERTIES_LIST_END_EXT list terminator for the list of 
                  properties

        CL_DEVICE_PARTITION_BY_AFFINITY_DOMAIN_EXT - Split the device into 
            smaller aggregate devices containing one or more compute units 
            that all share part of a cache hierarchy. The value accompanying 
            this property may be drawn from the following CL_AFFINITY_DOMAIN 
            list: 
        
            CL_AFFINITY_DOMAIN_NUMA_EXT - Split the device into sub-devices 
                comprised of compute units that share a NUMA band.

            CL_AFFINITY_DOMAIN_L4_CACHE_EXT - Split the device into sub-devices 
                comprised of compute units that share a level 4 data cache.

            CL_AFFINITY_DOMAIN_L3_CACHE_EXT - Split the device into sub-devices 
                comprised of compute units that share a level 3 data cache.

            CL_AFFINITY_DOMAIN_L2_CACHE_EXT - Split the device into sub-devices 
                comprised of compute units that share a level 2 data cache.

            CL_AFFINITY_DOMAIN_L1_CACHE_EXT - Split the device into sub-devices 
                comprised of compute units that share a level 1 data cache.

            CL_AFFINITY_DOMAIN_NEXT_FISSIONABLE_EXT - Split the device along the
                next fissionable CL_AFFINITY_DOMAIN.  The implementation shall 
                find the first level along which the device or sub-device may be
                further subdivided in the order NUMA, L4, L3, L2, L1, and 
                fission the device into sub-devices comprised of compute units 
                that share memory sub-systems at this level. The user may 
                determine what happened by calling 
                clGetDeviceInfo(CL_DEVICE_PARTITION_STYLE_EXT) on the 
                sub-devices.

            Example: To split a non-NUMA device along the outermost cache level 
            (if any), pass:

                { CL_DEVICE_PARTITION_BY_AFFINITY_DOMAIN_EXT, 
                  CL_AFFINITY_DOMAIN_NEXT_FISSIONABLE_EXT, 
                  CL_PROPERTIES_LIST_END_EXT }

        CL_PROPERTIES_LIST_END_EXT - A list terminator for a properties list.

    The following values may be returned by clCreateSubDevicesEXT:
    
    CL_SUCCESS - The command succeeded.

    CL_INVALID_VALUE - The properties key is unknown, or the indicated partition
        style (CL_DEVICE_PARTITION_BY_AFFINITY_DOMAIN_EXT, 
        CL_DEVICE_PARTITION_EQUALLY_EXT, CL_DEVICE_PARTITION_BY NAMES_EXT or 
        CL_DEVICE_PARTITION_BY COUNTS_EXT) is not supported for this device by 
        the implementation. On an OpenCL 1.1 implementation, these cases return 
        CL_INVALID_PROPERTY instead, to be consistent with clCreateContext 
        behavior.

    CL_INVALID_VALUE - num_entries is zero and out_devices is not NULL, or both 
        out_devices and num_devices are NULL.

    CL_DEVICE_PARTITION_FAILED_EXT - The indicated partition scheme is supported
        by the implementation, but the implementation can not further partition 
        the device in this way. For example, 
        CL_DEVICE_PARTITION_BY_AFFINITY_DOMAIN_EXT was requested, but all 
        compute units in in_device share the same cache at the level requested.

    CL_INVALID_PARTITION_COUNT_EXT - The total number of compute units requested
        exceeds CL_DEVICE_MAX_COMPUTE_UNITS, or the number of compute units for 
        any one sub-device is less than 1, or the number of sub-devices 
        requested exceeds CL_DEVICE_MAX_COMPUTE_UNITS.

    CL_INVALID_PARTITION_NAME_EXT - A compute unit name appearing in a name list
        following CL_DEVICE_PARTITION_BY NAMES_EXT is not in the range 
        [-1, number of compute units - 1].

    CL_INVALID_DEVICE - The in_device is not a valid device. The in_device is 
        not a device in context.


Clarification of Behavior of Existing Functions with Sub-Devices (Normative)

    In a style consistent with retain / release behavior of other OpenCL 
    objects, if a new object is created that references a cl_device_id, or 
    comes to reference it through an API such as clBuildProgram, the 
    cl_device_id shall be prevented from being destroyed until such time as 
    the new object no longer needs it.


Changes to Section 4.2 (Querying Devices)

    clGetDeviceIDs - Only root level devices shall be returned by this function.
        Devices returned by this function should not contain compute units that 
        alias compute units on another returned in the same function call. 
        Compute units in root level devices shall not alias compute units in 
        other root level devices.

    clGetDeviceInfo - If the device is a sub-device created by 
        clCreateSubDevicesEXT, then the value returned for 
        CL_DEVICE_MAX_COMPUTE_UNITS is the number of compute units in the 
        sub-device. The CL_DEVICE_VENDOR_ID may be different from the parent 
        device CL_DEVICE_VENDOR_ID, but should be the same for all devices and 
        sub-devices that can share a binary executable, such as that returned 
        from clGetProgramInfo(CL_PROGRAM_BINARIES). Other selectors such as 
        CL_DEVICE_GLOBAL_MEM_CACHE_SIZE may optionally change value to better 
        reflect the behavior of the sub-device in an implementation defined 
        manner.

        The following selectors are added for clGetDeviceInfo:

        CL_DEVICE_PARENT_DEVICE_EXT - a selector to get the cl_device_id for 
            the parent cl_device_id to which the sub-device belongs. 
            (Sub-division can be multi-level.) If the device is a root level 
            device, then it will return NULL.  
        
        CL_DEVICE_PARTITION_TYPES_EXT - a selector to get a list of supported 
            partition types for partitioning a device. The return type is an 
            array of cl_device partition property ext values drawn from the 
            following list:

            CL_DEVICE_PARTITION_BY_AFFINITY_DOMAIN_EXT
            CL_DEVICE_PARTITION_BY COUNTS_EXT
            CL_DEVICE_PARTITION_BY NAMES_EXT
            CL_DEVICE_PARTITION_EQUALLY_EXT 
            
            The implementation shall return at least one property from the above
            list.  However, when a partition style is found within this list, 
            the partition style is not required to work in every case. For 
            example, a device might support partitioning by affinity domain, but
            not along NUMA domains.

        CL_DEVICE_AFFINITY_DOMAINS_EXT - a selector to get a list of supported 
            affinity domains for partitioning the device using the 
            CL_DEVICE_PARTITION_BY_AFFINITY_DOMAIN_EXT partition style. The 
            return type is an array of cl_device_partition_property_ext values. 
            The values shall come from the list:

            CL_AFFINITY_DOMAIN_L1_CACHE_EXT 
            CL_AFFINITY_DOMAIN_L2_CACHE_EXT 
            CL_AFFINITY_DOMAIN_L3_CACHE_EXT 
            CL_AFFINITY_DOMAIN_L4_CACHE_EXT 
            CL_AFFINITY_DOMAIN_NUMA_EXT

            If no partition style is supported then the size of the returned 
            array is zero.  Even though a device has a NUMA, or particular 
            cache level, an implementation may elect not to provide fissioning 
            at that level.  
        
        CL_DEVICE_REFERENCE_COUNT_EXT Return the device 
            reference count.  The return type is cl_uint. If the device is a 
            root level device, a reference count of 1 is returned.

        CL_DEVICE_PARTITION_STYLE_EXT - a selector to get the 
            cl_device_partition_property_ext list used to create the sub-device.
            If the device is a root level device then a list consisting of 
            { CL_PROPERTIES_LIST_END_EXT} is returned. If the property on device
            creation was (CL_DEVICE_PARTITION BY_AFFINITY_DOMAIN_EXT, 
            CL_AFFINITY_DOMAIN_NEXT_FISSIONABLE) then 
            CL_AFFINITY_DOMAIN_NEXT_FISSIONABLE will be replaced by the symbol 
            representing the actual CL_AFFINITY DOMAIN used 
            (e.g. CL_AFFINITY_DOMAIN_NUMA). The returned value is an array of 
            cl_device_partition_property_ext. The length of the array is 
            obtained from the size returned by the param size value ret 
            parameter to the function.

Changes to Section 4.3 (Contexts)

    clCreateContext - The new cl_context only owns a reference to the devices 
        explicitly passed to it at creation. It does not reference sub-devices 
        of those devices unless they were also passed in the device list at 
        creation time.

    clCreateContextFromType - Only root level devices are used to create the 
        context.  The context does not reference sub-devices created from those 
        devices.

    clGetContextInfo - Only the device list passed into clCreateContext is 
        returned by CL_CONTEXT_DEVICES.  Additional sub-devices usable in the 
        context, because their parent devices were used to create the devices, 
        are not returned here.

Changes to Section 5.1 (Command Queues)

    clCreateCommandQueue - This call will return CL_INVALID_DEVICE if the device
        or one of its ancestors was not used to create the context. On success,
        the cl_command_queue that is created for a sub-device will attempt to 
        execute any work enqueued on it on just the sub-device.

    clGetCommandQueueInfo - If the command queue was created with a sub-device,
        the sub-device is returned for the CL_QUEUE_DEVICE selector.

Changes to Section 5.4.1 (Creating Program Objects)

    clCreateProgramWithBinary - This call will return CL_INVALID_DEVICE if a 
        device or one of its ancestors was not used to create the programs 
        context.

Changes to Section 5.4.2 (Building Program Executables)

    clBuildProgram - This call will return CL_INVALID_DEVICE if the device or 
        one of its ancestors was not used to create the programs context.  
        clBuildProgram must be called for each device or sub-device that will 
        be used with the program. This function will return CL_INVALID_DEVICE 
        if the device was not used to create the program and the program was 
        created with clCreateProgramWithBinary.  Implementations should 
        eliminate redundant compilation where devices and sub-devices can share
        a common binary.

Changes to Section 5.4.5 (Program Object Queries)

    clGetProgramInfo - Behavior depends on how the program was created:

        clCreateProgramWithSource - CL_PROGRAM_DEVICES returns the union of the 
            set of devices used to create the cl_context to which the program 
            belongs and the set of devices for which clBuildProgram() has 
            succeeded on this program.
        
        clCreateProgramWithBinary - CL_PROGRAM_DEVICES returns the union of the 
            set of devices used to create the program and the set of devices 
            for which clBuildProgram() has succeeded on this program.  
        
        In each case, CL_PROGRAM_BINARIES and CL_PROGRAM_BINARY_SIZES returns 
        the same list in the same order, except that pointers to compiled 
        binaries or their sizes are returned instead, respectively. The user is 
        responsible for preventing the obvious race condition here, in which 
        clBuildProgram() succeeds on more devices between when 
        CL_PROGRAM_DEVICES, CL_PROGRAM_BINARIES and CL_PROGRAM_BINARY SIZES are 
        requested.