<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">

<rfc category="info" docName="sp-request-reply-01">

  <front>

    <title abbrev="Request/Reply SP">
    Request/Reply Scalability Protocol
    </title>

    <author fullname="Martin Sustrik" initials="M." role="editor"
            surname="Sustrik">
      <organization>GoPivotal Inc.</organization>
      <address>
        <email>msustrik@gopivotal.com</email>
      </address>
    </author>

    <date month="August" year="2013" />

    <area>Applications</area>
    <workgroup>Internet Engineering Task Force</workgroup>

    <keyword>Request</keyword>
    <keyword>Reply</keyword>
    <keyword>REQ</keyword>
    <keyword>REP</keyword>
    <keyword>stateless</keyword>
    <keyword>service</keyword>
    <keyword>SP</keyword>

    <abstract>
      <t>This document defines a scalability protocol used for distributing
         processing tasks among arbitrary number of stateless processing nodes
         and returning the results of the processing.</t>
    </abstract>

  </front>

  <middle>

    <section title = "Introduction">

      <t>One of the most common problems in distributed applications is how to
         delegate a work to another processing node and get the result back to
         the original node. In other words, the goal is to utilise the CPU
         power of a remote node.</t>

      <t>There's a wide range of RPC systems addressing the problem, however,
         instead of relying on simple RPC algorithm, we will aim at solving a
         more general version of the problem. First, we want to issue processing
         requests from multiple clients, not just a single one. Second, we want
         to distribute the tasks to any number processing nodes instead of a
         single one so that the processing can be scaled up by adding new
         processing nodes as necessary.</t>

      <t>Solving the generalised problem requires that the algorithm
         executing the task in question -- also known as "service" -- is
         stateless.</t>

      <t>To put it simply, the service is called "stateless" when there's no
         way for the user to distinguish whether a request was processed by
         one instance of the service or another one.</t>

      <t>So, for example, a service which accepts two integers and multiplies
         them is stateless. Request for "2x2" is always going to produce "4",
         no matter what instance of the service have computed it.</t>

      <t>Service that accepts empty requests and produces the number
         of requests processed so far (1, 2, 3 et c.), on the other hand, is
         not stateless. To prove it you can run two instances of the service.
         First reply, no matter which instance produces it is going to be 1.
         Second reply though is going to be either 2 (if processed by the same
         instance as the first one) or 1 (if processed by the other instance).
         You can distinguish which instance produced the result. Thus,
         according to the definition, the service is not stateless.</t>

      <t>Despite the name, being "stateless" doesn't mean that the service has
         no state at all. Rather it means that the service doesn't retain any
         business-logic-related state in-between processing two subsequent
         requests. The service is, of course, allowed to have state while
         processing a single request. It can also have state that is unrelated
         to its business logic, say statistics about the processing that are
         used for administrative purposes and never returned to the clients.</t>

      <t>Also note that "stateless" doesn't necessarily mean "fully
         deterministic". For example, a service that generates random numbers is
         non-deterministic. However, the client, after receiving a new random
         number cannot tell which instance has produced it, thus, the service
         can be considered stateless.</t>

      <t>While stateless services are often implemented by passing the entire
         state inside the request, they are not required to do so. Especially
         when the state is large, passing it around in each request may be
         impractical. In such cases, it's typically just a reference to the
         state that's passed in the request, such as ID or path. The state
         itself can then be retrieved by the service from a shared database,
         a network file system or similar storage mechanism.</t>
         
      <t>Requiring services to be stateless serves a specific purpose.
         It allows for using any number of service instances to handle
         the processing load. After all, the client won't be able to tell the
         difference between replies from instance A and replies from instance B.
         You can even start new instances on the fly and get away with it.
         The client still won't be able to tell the difference. In other
         words, statelessness is a prerequisite to make your service cluster
         fully scalable.</t>

      <t>Once it is ensured that the service is stateless there are several
         topologies for a request/reply system to form. What follows are
         the most common:

        <list style = "numbers">
          <t>One client sends a request to one server and gets a reply.
             The common RPC scenario.</t>
          <t>Many clients send requests to one server and get replies. The
             classic client/server model. Think of a database server and
             database clients. Alternatively think of a messaging broker and
             messaging clients.</t>
          <t>One client send requests to many servers and gets replies.
             The load-balancer model. Think of HTTP load balancers.</t>
          <t>Many clients send requests to be processed by many servers.
             The "enterprise service bus" model. In the simplest case the bus
             can be implemented as a simple hub-and-spokes topology. In complex
             cases the bus can span multiple physical locations or multiple
             oraganisations with intermediate nodes at the boundaries connecting
             different parts of the topology.</t>
        </list>

      </t>

      <t>In addition to distributing tasks to processing nodes, request/reply
         model comes with full end-to-end reliability. The reliability guarantee
         can be defined as follows: As long as the client is alive and there's
         at least one server accessible from the client, the task will
         eventually get processed and the result will be delivered back to
         the client.</t>

      <t>End-to-end reliability is achieved, similar to TCP, by re-sending the
         request if the client believes the original instance of the request
         has failed. Typically, request is believed to have failed when there's
         no reply received within a specified time.</t>

      <t>Note that, unlike with TCP, the reliability algorithm is resistant to
         a server failure. Even if server fails while processing a request, the
         request will be re-sent and eventually processed by a different
         instance of the server.</t>

      <t>As can be seen from the above, one request may be processed multiple
         times. For example, reply may be lost on its way back to the client.
         Client will assume that the request was not processed yet, it will
         resend it and thus cause duplicit execution of the task.</t>

      <t>Some applications may want to prevent duplicit execution of tasks. It
         often turns out that hardening such applications to be idempotent is
         relatively easy as they already possess the tools to do so. For
         example, a payment processing server already has access to a shared
         database which it can use to verify that the payment with specified ID
         was not yet processed.</t>

      <t>On the other hand, many applications don't care about occasional
         duplicitly processed tasks. Therefore, request/reply protocol does not
         require the service to be idempotent. Instead, the idempotancy issue
         is left to the user to decide on.</t>

      <t>Finally, it should be noted that this specification discusses several
         features that are of little use in simple topologies and are rather
         aimed at large, geographically or organisationally distributed
         topologies. Features like channel prioritisation and loop avoidance
         fall into this category.</t>

    </section>

    <section title = "Underlying protocol">

      <t>The request/reply protocol can be run on top of any SP mapping,
         such as, for example, SP TCP mapping.</t>

      <t>Also, given that SP protocols describe the behaviour of entire
         arbitrarily complex topology rather than of a single node-to-node
         communication, several underlying protocols can be used in parallel.
         For example, a client may send a request via WebSocket, then, on the
         edge of the company network an intermediary node may retransmit it
         using TCP et c.</t>

      <figure>
        <artwork>
+---+  WebSocket  +---+    TCP    +---+
|   |-------------|   |-----------|   |
+---+             +---+           +---+
                   | |
     +---+   IPC   | |  SCTP  +---+    DCCP   +---+
     |   |---------+ +--------|   |-----------|   |
     +---+                    +---+           +---+
        </artwork>
      </figure>

    </section>

    <section title = "Overview of the algorithm">

      <t>Request/reply protocol defines two different endpoint types:
         The requester or REQ (the client) and the replier or REP (the
         service).</t>

      <t>REQ endpoint can be connected only to a REP endpoint. REP endpoint
         can be connected only to the REQ endpoint. If the underlying protocol
         indicates that there's an attempt to create a channel to an
         incompatible endpoint, the channel MUST NOT be used. In the case of
         TCP mapping, for example, the underlying TCP connection MUST
         be closed.</t>

      <t>When creating more complex topologies, REQ and REP endpoints are
         paired in the intermediate nodes to form a forwarding component,
         so called "device". Device receives requests from the REP endpoint
         and forwards them to the REQ endpoint. At the same time it receives
         replies from the REQ endpoint and forwards them to the REP
         endpoint:</t>

      <figure>
        <artwork>
                --- requests --&gt;

+-----+   +-----+-----+   +-----+-----+   +-----+
|     |--&gt;|     |     |--&gt;|     |     |--&gt;|     |
| REQ |   | REP | REQ |   | REP | REQ |   | REP |
|     |&lt;--|     |     |&lt;--|     |     |&lt;--|     |
+-----+   +-----+-----+   +-----+-----+   +-----+

                &lt;-- replies ---
        </artwork>
      </figure>

      <t>Using devices, arbitrary complex topologies can be built. The rest
         of this section explains how are the requests routed through a topology
         towards processing nodes and how are replies routed back from
         processing nodes to the original clients, as well as how the
         reliability is achieved.</t>

      <t>The idea for routing requests is to implement a simple coarse-grained
         scheduling algorithm based on pushback capabilities of the underlying
         transport.</t>

      <t>The algorithm works by interpreting pushback on a particular channel
         as "the part of topology accessible through this channel is busy at
         the moment and doesn't accept any more requests."</t>

      <t>Thus, when a node is about to send a request, it can choose to send
         it only to one of the channels that don't report pushback at the
         moment. To implement approximately fair distibution of the workload
         the node choses a channel from that pool using the round-robin
         algorithm.</t>

      <t>As for delivering replies back to the clients, it should be understood
         that the client may not be directly accessible (say using TCP/IP) from
         the processing node. It may be beyond a firewall, have no static IP
         address et c. Furthermore, the client and the processing may not even
         speak the same transport protocol -- imagine client connecting to the
         topology using WebSockets and processing node via SCTP.</t>

      <t>Given the above, it becomes obvious that the replies must be routed
         back through the existing topology rather than directly. In fact,
         request/reply topology may be thought of as an overlay network on the
         top of underlying transport mechanisms.</t>

      <t>As for routing replies within the request/topology, it is designed in
         such a way that each reply contains the whole routing path, rather
         than containing just the address of destination node, as is the case
         with, for example, TCP/IP.</t>

      <t>The downside of the design is that replies are a little bit longer
         and that is in intermediate node gets restarted, all the requests
         that were routed through it will fail to complete and will have to be
         resent by request/reply end-to-end reliability mechanism.</t>

      <t>The upside, on the other hand, is that the nodes in the topology don't
         have to maintain any routing tables beside the simple table of
         adjacent channels along with thier IDs. There's also no need for any
         additional protocols for distributing routing information within
         the topology.</t>

      <t>The most important reason for adopting the design though is that
         there's no propagation delay and any nodes becomes accessible
         immediately after it is started. Given that some nodes in the topology
         may be extremely short-lived this is a crucial requirement. Imagine
         a database client that sends a query, reads the result and terminates.
         It makes no sense to delay the whole process till the routing tables
         are synchronised between the client and the server.</t>

      <t>The algorithm thus works as follows: When request is routed from the
         client to the processing node, every REP endpoint determines which
         channel it was received from and adds the ID of the channel to the
         request. Thus, when the request arrives at the ultimate processing node
         it already contains a full backtrace stack, which in turn contains
         all the info needed to route a message back to the original client.</t>

      <t>After processing the request, the processing node attaches the
         backtrace stack from the request to the reply and sends it back
         to the topology. At that point every REP endpoint can check the
         traceback and determine which channel it should send the reply to.</t>

      <t>In addition to routing, request/reply protocol takes care of
         reliability, i.e. ensures that every request will be eventually
         processed and the reply will be delivered to the user, even when
         facing failures of processing nodes, intermediate nodes and network
         infrastructure.</t>

      <t>Reliability is achieved by simply re-sending the request, if the reply
         is not received with a certain timeframe. To make that algorithm
         work flawlessly, the client has to be able to filter out any stray
         replies (delayed replies for the requests that we've already received
         reply to).</t>

      <t>The client thus adds an unique request ID to the request. The ID gets
         copied from the request to the reply by the processing node. When the
         reply gets back to the client, it can simply check whether the request
         in question is still being processed and if not so, it can ignore
         the reply.</t>

      <t>To implement all the functionality described above, messages (both
         requests and replies have the following format:</t>

      <figure>
        <artwork>
+-+------------+-+------------+   +-+------------+-------------+
|0| Channel ID |0| Channel ID |...|1| Request ID |   payload   |
+-+------------+-+------------+   +-+------------+ ------------+
        </artwork>
      </figure>

      <t>Payload of the message is preceded by a stack of 32-bit tags. The most
         significant bit of each tag is set to 0 except for the very last tag.
         That allows the algorithm to find out where the tags end and where
         the message payload begins.</t>

      <t>As for the reamining 31 bits, they are either request ID (in the last
         tag) or a channel ID (in all the remaining tags). The first channel ID
         is added and processed by the REP endpoint closest to the processing
         node. The last channel ID is added and processed by the REP endpoint
         closest to the client.</t>

      <t>Following picture shows an example of request saying "Hello" being
         routed from the client through two intermediate nodes to the
         processing node and the reply "World" being routed back. It shows
         what messages are passed over the network at each step of the
         process:</t>

      <figure>
        <artwork>
                        client
                  Hello    |    World
                   |    +-----+    ^
                   |    | REQ |    |
                   V    +-----+    |
            1|823|Hello    |    1|823|World
                   |    +-----+    ^
                   |    | REP |    |
                   |    +-----+    |
                   |    | REQ |    |
                   V    +-----+    |
      0|299|1|823|Hello    |    0|299|1|823|World
                   |    +-----+    ^
                   |    | REP |    |
                   |    +-----+    |
                   |    | REQ |    |
                   V    +-----+    |
0|446|0|299|1|823|Hello    |    0|446|0|299|1|823|World
                   |    +-----+    ^
                   |    | REP |    |
                   V    +-----+    |
                  Hello    |    World
                        service
        </artwork>
      </figure>

    </section>

    <section title = "Hop-by-hop vs. End-to-end">

      <t>All endpoints implement so called "hop-by-hop" functionality. It's
         the functionality concerned with sending messages to the immediately
         adjacent components and receiving messages from them.</t>

      <t>In addition to that, the endpoints on the edge of the topology
         implement so called "end-to-end" functionality that is concerned
         with issues such as, for example, reliability.</t>

      <figure>
        <artwork>
                   end to end
   +-----------------------------------------+
   |                                         |
+-----+   +-----+-----+   +-----+-----+   +-----+
|     |--&gt;|     |     |--&gt;|     |     |--&gt;|     |
| REQ |   | REP | REQ |   | REP | REQ |   | REP |
|     |&lt;--|     |     |&lt;--|     |     |&lt;--|     |
+-----+   +-----+-----+   +-----+-----+   +-----+
   |         |     |         |     |         |
   +---------+     +---------+     +---------+
   hop by hop      hop by hop      hop by hop
        </artwork>
      </figure>

      <t>To make an analogy with the TCP/IP stack, IP provides hop-by-hop
         functionality, i.e. routing of the packets to the adjacent node,
         while TCP implements end-to-end functionality such resending of
         lost packets.</t>

      <t>As a rule of thumb, raw hop-by-hop endpoints are used to build
         devices (intermediary nodes in the topology) while end-to-end
         endpoints are used directly by the applications.</t>

      <t>To prevent confusion, the specification of the endpoint behaviour
         below will discuss hop-by-hop and end end-to-end functionality in
         separate chapters.</t>

    </section>

    <section title = "Hop-by-hop functionality">

      <section title = "REQ endpoint">

        <t>The REQ endpoint is used by the user to send requests to the
           processing nodes and receive the replies afterwards.</t>

        <t>When user asks REQ endpoint to send a request, the endpoint should
           send it to one of the associated outbound channels (TCP connections
           or similar). The request sent is exactly the message supplied by
           the user. REQ socket MUST NOT modify an outgoing request in any
           way.</t>

        <t>If there's no channel to send the request to, the endpoint won't send
           the request and MUST report the backpressure condition to the user.
           For example, with BSD socket API, backpressure is reported as EAGAIN
           error.</t>

        <t>If there are associated channels but none of them is available for
           sending, i.e. all of them are already reporting backpressure, the
           endpoint won't send the message and MUST report the backpressure
           condition to the user.</t>

        <t>Backpressure is used as a means to redirect the requests from the
           congested parts of the topology to to the parts that are still
           responsive. It can be thought of as a crude scheduling algorithm.
           However crude though, it's probably still the  best you can get
           without knowing estimates of execution time for individual tasks,
           CPU capacity of individual processing nodes et c.</t>

        <t>Alternatively, backpressure can be thought of as a congestion control
           mechanism. When all available processing nodes are busy, it slows
           down the client application, i.e. it prevents the user from sending
           any more requests.</t>

        <t>If the channel is not capable of reporting backpressure (e.g. DCCP)
           the endpoint SHOULD consider it as always available for sending new
           request. However, such channels should be used with care as when the
           congestion hits they may suck in a lot of requests just to discard
           them silently and thus cause re-transmission storms later on. The
           implementation of the REQ endpoint MAY choose to prohibit the use
           of such channels altogether.</t>

        <t>When there are multiple channels available for sending the request
           endpoint MAY use any prioritisation mechanism to decide which channel
           to send the request to. For example, it may use classic priorities
           attached to channels and send message to the channel with the highest
           priority. That allows for routing algorithms such as: "Use local
           processing nodes if any are available. Send the requests to remote
           nodes only if there are no local ones available." Alternatively,
           the endpoint may implement weighted priorities ("send 20% of the
           request to node A and 80% to node B). The endpoint also may not
           implement any prioritisation strategy and treat all channels as
           equal.</t>

        <t>Whatever the case, two rules must apply.</t>

        <t>First, by default the priority settings for all channels MUST be
           equal. Creating a channel with different priority MUST be triggered
           by an explicit action by the user.</t>

        <t>Second, if there are several channels with equal priority, the
           endpoint MUST distribute the messages among them in fair fashion
           using round-robin algorithm. The round-robin implementation MUST also
           take care not to become unfair when new channels are added or old
           ones are removed on the fly.</t>

        <t>As for incoming messages, i.e. replies, REQ endpoint MUST fair-queues
           them. In other words, if there are replies available on several
           channels, it MUST receive them in a round-robin fashion. It mast also
           take care not to compromise the fairness when new channels are
           added or old ones removed.</t>

        <t>In addition to providing basic fairness, the goal of fair-queueing is
           to prevent DoS attacks where a huge stream of fake replies from one
           channel would be able to block the real replies coming from different
           channels. Fair queueing ensures that messages from every channel are
           received at approximately the same rate. That way, DoS attack can
           slow down the system but it can't entirely block it.</t>

        <t>Incoming replies MUST be handed to the user exactly as they were
           received. REQ endpoint MUST not modify the replies in any way.</t>

      </section>

      <section title = "REP endpoint">

        <t>REP endpoint is used to receive requests from the clients and send
           replies back to the clients.</t>

        <t>First of all, REP socket is responsible for assigning unique 31-bit
           channel IDs to the individual associated channels.</t>

        <t>First ID assigned MUST be random. Next is computed by adding 1 to
           the previous one with potential overflow to 0.</t>

        <t>The implementation MUST ensure that the random number is different
           each time the endpoint is re-started, the process that contains
           it is restarted or similar. So, for example, using pseudo-random
           generator with a constant seed won't do.</t>

        <t>The goal of the algorithm is to the spread of possible channel ID
           values and thus minimise the chance that a reply is routed to an
           unrelated channel, even in the face of intermediate node
           failures.</t>

        <t>When receiving a message, REP endpoint MUST fair-queue among the
           channels available for receiving. In other words it should
           round-robin among such channels and receive one request from
           a channel at a time. It MUST also implement the round-robin
           algorithm is such a way that adding or removing channels don't
           break its fairness.</t>

        <t>In addition to guaranteeing basic fairness in access to computing
           resources the above algorithm makes it impossible for a malevolent
           or misbehaving client to completely block the processing of requests
           from other clients by issuing steady stream of requests.</t>

        <t>After getting hold on the request, the REP socket should prepend it
           by 32 bit value, consisting of 1 bit set to 0 followed by the 31-bit
           ID of the channel the request was received from. The extended request
           will be then handed to the user.</t>

        <t>The goal of adding the channel ID to the request is to be able to
           route the reply back to the original channel later on. Thus, when
           the user sends a reply, endpoint strips first 32 bits off and uses
           the value to determine where it is to be routed.</t>

        <t>If the reply is shorter than 32 bits, it is malformed and
           the endpoint MUST ignore it. Also, if the most relevant bit of the
           32-bit value isn't set to 0, the reply is malformed and MUST
           be ignored.</t>

        <t>Otherwise, the endpoint checks whether its table of associated
           channels contains the channel with a corresponding ID. If so, it
           sends the reply (with first 32 bits stripped off) to that channel.
           If the channel is not found, the reply MUST be dropped. If the
           channel is not available for sending, i.e. it is applying
           backpressure, the reply MUST be dropped.</t>

        <t>Note that when the reply is unroutable two things might have
           happened. Either there was some kind of network disruption, in which
           case the request will be re-sent later on, or the original client
           have failed or been shut down. In such case the request won't be
           resent, however, it doesn't really matter because there's no one to
           deliver the reply to any more anyway.</t>

        <t>Also note that unlike with the requests there's no pushback applied
           to the replies. The replies are simply dropped in the case of
           pushback. The reason for this behaviour is that if the endpoint
           blocked and waited for the channel be become available, all the
           subsequent replies, possibly destined for different, unblocked
           channels would be blocked in the meantime anyway. That would allow
           for a DoS attack by simply firing a lot of requests and not receiving
           the replies.</t>

      </section>
   
    </section>

    <section title = "End-to-end functionality">

      <t>End-to-end functionality is built on top of hop-to-hop functionality.
         Thus, an endpoint on the edge of a topology contains all the
         hop-by-hop functionality, but also implements additional
         functionality of its own. This end-to-end functionality acts
         basically as a user of the underlying hop-by-hop functionality.</t>

      <section title = "REQ endpoint">

        <t>End-to-end functionality for REQ sockets is concerned with re-sending
           the requests in case of failure and with filtering out stray or
           outdated replies.</t>

        <t>To be able to do the latter, the endpoint must tag the requests with
           unique 31-bit request IDs. First request ID is picked at random. All
           subsequent request IDs are generated by adding 1 to the last request
           ID and possibly overflowing to 0.</t>

        <t>To improve robustness of the system, the implementation MUST ensure
           that the random number is different each time the endpoint, the
           process or the machine is restarted. Pseudo-random generator with
           fixed seed won't do.</t>

        <t>When user asks the endpoint to send a message, the endpoint prepends
           a 32-bit value to the message, consisting of a single bit set to 1
           followed by a 31-bit request ID and passes it on in a standard
           hop-by-hop way.</t>

        <t>If the hop-by-hop layer reports pushback condition, the end-to-end
           layer considers the request unsent and MUST report pushback condition
           to the user.</t>

        <t>If the request is successfully sent, the endpoint stores the request
           including its request ID, so that it can be resent later on if
           needed. At the same time it sets up a timer to trigger the
           re-transimission in case the reply is not received within a specified
           timeout. The user MUST be allowed to specify the timeout interval.
           The default timeout interval must be 60 seconds.</t>

        <t>When a reply is received from the underlying hop-by-hop
           implementation, the endpoint should strip off first 32 bits from
           the reply to check whether it is a valid reply.</t>

        <t>If the reply is shorter than 32 bits, it is malformed and the
           endpoint MUST ignore it. If the most significant bit of the 32-bit
           value is set to 0, the reply is malformed and MUST be ignored.</t>

        <t>Otherwise, the endpoint should check whether the request ID in
           the reply matches any of the request IDs of the requests being
           processed at the moment. If not so, the reply MUST be ignored.
           It is either a stray message or a duplicate reply.</t>

        <t>Please note that the endpoint can support either one or more
           requests being processed in parallel. Which one is the case depends
           on the API exposed to the user and is not part of this
           specification.</t>

        <t>If the ID in the reply matches one of the requests in progress, the
           reply MUST be passed to the user (with the 32-bit prefix stripped
           off). At the same time the stored copy of the original request as
           well as re-transmission timer must be deallocated.</t>

        <t>Finally, REQ endpoint MUST make it possible for the user to cancel
           a particular request in progress. What it means technically is
           deleting the stored copy of the request and cancelling the associated
           timer. Thus, once the reply arrives, it will be discarded by the
           algorithm above.</t>

        <t>The cancellation allows, for example, the user to time out a request.
           They can simply post a request and if there's no answer in specific
           timeframe, they can cancel it.</t>

      </section>

      <section title = "REP endpoint">

        <t>End-to-end functionality for REP endpoints is concerned with turning
           requests into corresponding replies.</t>

        <t>When user asks to receive a request, the endpoint gets next request
           from the hop-by-hop layer and splits it into the traceback stack and
           the message payload itself. The traceback stack is stored and the
           payload is returned to the user.</t>

        <t>The algorithm for splitting the request is as follows: Strip 32 bit
           tags from the message in one-by-one manner. Once the most significant
           bit of the tag is set, we've reached the bottom of the traceback
           stack and the splitting is done. If the end of the message is reached
           without finding the bottom of the stack, the request is malformed and
           MUST be ignored.</t>

        <t>Note that the payload payload produced by this procedure is the same
           as the request payload sent by the original client.</t>

        <t>Once the user processes the request and sends the reply, the endpoint
           prepends the reply with the stored traceback stack and sends it on
           using the hop-by-hop layer. At that point the stored traceback stack
           MUST be deallocated.</t>

        <t>Additionally, REP endpoint MUST support cancelling any request being
           processed at the moment. What it means, technically, is that
           state associated with the request, i.e. the traceback stack stored
           by the endpoint is deleted and reply to that particular
           request is never sent.</t>

        <t>The most important use of cancellation is allowing the service
           instances to ignore malformed requests. If the application-level
           part of the request doesn't conform to the application protocol
           the service can simply cancel the request. In such case the reply
           is never sent. Of course, if application wants to send an
           application-specific error massage back to the client it can do so
           by not cancelling the request and sending a regular reply.</t>

      </section>

    </section>

    <section title = "Loop avoidance">

      <t>It may happen that a request/reply topology contains a loop. It becomes
         increasingly likely as the topology grows out of scope of a single
         organisation and there are multiple administrators involved
         in maintaining it. Unfortunate interaction between two perfectly
         legitimate setups can cause loop to be created.</t>

      <t>With no additional guards against the loops, it's likely that
         requests will be caugth inside the loop, rotating there forever,
         each message gradually growing in size as new prefixes are added to it
         by each REP endpoint on the way. Eventually, a loop can cause
         congestion and bring the whole system to a halt.</t>

      <t>To deal with the problem REQ endpoints MUST check the depth of the
         traceback stack for every outgoing request and discard any requests
         where it exceeds certain threshold. The threshold should be defined
         by the user. The default value is suggested to be 8.</t>

    </section>

    <section anchor="IANA" title="IANA Considerations">
      <t>New SP endpoint types REQ and REP should be registered by IANA. For
         now, value of 16 should be used for REQ endpoints and value of 17 for
         REP endpoints.</t>
    </section>

    <section anchor="Security" title="Security Considerations">
      <t>The mapping is not intended to provide any additional security to the
         underlying protocol. DoS concerns are addressed within
         the specification.</t>
    </section>

  </middle>

</rfc>