Ucx Note

Basic

• typedef struct ucp_context∗ ucp_context_h

UCP application context (or just a context) is an opaque handle that holds a UCP communication instance's global information. It represents a single UCP communication instance. The communication instance could be an OS process (an application) that uses UCP library. This global information includes communication resources, end-points, memory, temporary file storage, and other communication information directly associated with a specific UCP instance. The context also acts as an isolation mechanism, allowing resources associated with the context to manage multiple concurrent communication instances. For example, users using both MPI and OpenSHMEM sessions simultaneously can isolate their communication by allocating and using separate contexts for each of them. Alternatively, users can share the communication resources (memory, network resource context, etc.) between them by using the same application context. A message sent or a RMA operation performed in one application context cannot be received in any other application context.

• typedef struct ucp_worker∗ ucp_worker_h

UCP worker is an opaque object representing the communication context. The worker represents an instance of a local communication resource and the progress engine associated with it. The progress engine is a construct that is responsible for asynchronous and independent progress of communication directives. The progress engine could be implemented in hardware or software. The worker object abstracts an instance of network resources such as a host channel adapter port, network interface, or multiple resources such as multiple network interfaces or communication ports. It could also represent virtual communication resources that are defined across multiple devices. Although the worker can represent multiple network resources, it is associated with a single UCX application context. All communication functions require a context to perform the operation on the dedicated hardware resource(s** and an endpoint to address the destination.

Note: Worker are parallel "threading points** that an upper layer may use to optimize concurrent communications.

• typedef struct ucp_params ucp_params_t

The structure defines the parameters that are used for UCP library tuning during UCP library initialization.

Note: UCP library implementation uses the features parameter to optimize the library functionality that minimize memory footprint. For example, if the application does not require send/receive semantics UCP library may avoid allocation of expensive resources associated with send/receive queues.

• struct ucp_params

The structure defines the parameters that are used for UCP library tuning during UCP library initialization.

Data Fields: field_mask, features, request_size, request_init, request_cleanup, tag_sender_mask, mt_workers_shared, estimated_num_eps, estimated_num_ppn, name

• ucp_init()

1
2
3
4

static ucs_status_t ucp_init (
const ucp_params_t ∗ params,
const ucp_config_t ∗ config,
ucp_context_h ∗ context_p ) [inline], [static]

This routine creates and initializes a UCP application context.

Warning This routine must be called before any other UCP function call in the application. This routine checks API version compatibility, then discovers the available network interfaces, and initializes the network resources required for discovering of the network and memory related devices. This routine is responsible

for initialization all information required for a particular application scope, for example, MPI application, OpenSHMEM application, etc.

Note

1. Higher level protocols can add additional communication isolation, as MPI does with it's communicator object. A single communication context may be used to support multiple MPI communicators.
2. The context can be used to isolate the communication that corresponds to different protocols. For exam- ple, if MPI and OpenSHMEM are using UCP to isolate the MPI communication from the OpenSHMEM communication, users should use different application context for each of the communication libraries.

• ucp_conn_request_h

typedef struct ucp_conn_request∗ ucp_conn_request_h

A server-side handle to incoming connection request. Can be used to create an endpoint which connects back to the client.

• ucp_listener_h

typedef struct ucp_listener∗ ucp_listener_h

The listener handle is an opaque object that is used for listening on a specific address and accepting connections from clients.

• ucp_ep_h

typedef struct ucp_ep∗ ucp_ep_h

The endpoint handle is an opaque object that is used to address a remote worker. It typically provides a description of source, destination, or both. All UCP communication routines address a destination with the endpoint handle. The endpoint handle is associated with only one UCP context. UCP provides the endpoint create routine to create the endpoint handle and the destroy routine to destroy the endpoint handle.

• ucp_am_handler_param_t

typedef struct ucp_am_handler_param ucp_am_handler_param_t

• `struct ucp_am_handler_param**

Data Fields: field_mask, id, flags, cb, arg

• ucp_worker_create()

1
2
3
4

ucs_status_t ucp_worker_create (
ucp_context_h context,
const ucp_worker_params_t ∗ params,
ucp_worker_h ∗ worker_p )

This routine allocates and initializes a worker object. Each worker is associated with one and only one application context. In the same time, an application context can create multiple workers in order to enable concurrent access to communication resources. For example, application can allocate a dedicated worker for each application thread, where every worker can be progressed independently of others.

Note

The worker object is allocated within context of the calling thread

• ucp_cleanup()

void ucp_cleanup (
ucp_context_h context_p )

This routine finalizes and releases the resources associated with a UCP application context.

• ucp_listener_params_t

typedef struct ucp_listener_params ucp_listener_params_t

This structure defines parameters for ucp_listener_create, which is used to listen for incoming client/server connec- tions.

• ucp_listener_query()

ucs_status_t ucp_listener_query (
ucp_listener_h listener,
ucp_listener_attr_t ∗ attr )

This routine fetches information about the listener.

• ucp_stream_send_nbx()

ucs_status_ptr_t ucp_stream_send_nbx (
ucp_ep_h ep,
const void ∗ buffer,
size_t count,
const ucp_request_param_t ∗ param )

This routine sends data that is described by the local address buffer, size count object to the destination endpoint ep. The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed

The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine returns UCS_OK.

Note: The user should not modify any part of the buffer after this operation is called, until the operation completes.

example

init context

Initialize the UCP context and worker.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

/**
 * Initialize the UCP context and worker.
 */
static int init_context(ucp_context_h *ucp_context, ucp_worker_h *ucp_worker,
                        send_recv_type_t send_recv_type)
{
    /* UCP objects */
    ucp_params_t ucp_params;
    ucs_status_t status;
    int ret = 0;

    memset(&ucp_params, 0, sizeof(ucp_params));

    /* UCP initialization */
    ucp_params.field_mask = UCP_PARAM_FIELD_FEATURES;

    if (send_recv_type == CLIENT_SERVER_SEND_RECV_STREAM) {
        ucp_params.features = UCP_FEATURE_STREAM;
    } else if (send_recv_type == CLIENT_SERVER_SEND_RECV_TAG) {
        ucp_params.features = UCP_FEATURE_TAG;
    } else {
        ucp_params.features = UCP_FEATURE_AM;
    }

    status = ucp_init(&ucp_params, NULL, ucp_context);
    if (status != UCS_OK) {
        fprintf(stderr, "failed to ucp_init (%s)\n", ucs_status_string(status));
        ret = -1;
        goto err;
    }

    ret = init_worker(*ucp_context, ucp_worker);
    if (ret != 0) {
        goto err_cleanup;
    }

    return ret;

err_cleanup:
    ucp_cleanup(*ucp_context);
err:
    return ret;
}