FIXNETIX FPGA TECHNOLOGY

IX-ECUTE NANOSECOND TRADING SYSTEM

In this paper we examine the methodology employed by Fixnetix for the iX-eCute FPGAbased pre-trade risk control system. Important differences between the Fixnetix implementation and competing architectures are evaluated as an insight into why Fixnetix has the competitive edge and will remain the dominant force in FPGA trading systems.

THE RACE TO ZERO.

FPGA technology has long been considered the panacea of software development, due to its efficiency and deterministic performance. However, a number of negative perceptions persist as to the technologies applicability in the Financial Technology (FinTech) world and it is fair to say that a large amount of up-front effort is required, in order to take a design to production and avoid key pitfalls. Fixnetix has been involved in FPGA development for over 3 years and has a vast array of developer experience with backgrounds, ranging from military, telecoms, consumer electronics and academia. However, a strong talent pool is not enough to develop world class FPGA trading systems in isolation. The correct balance of design, architecture, simulation, verification practical trading system know-how all culminate in FPGA systems seamlessly integrate into legacy systems of investment banks hedge funds, while providing industry leading capabilities performance with recognized regulatory compliance. and that and and

WHAT IS AN FPGA?
FPGA stands for Field Programmable Gate Array. Internally, the FPGA can be thought of as a series of overlaid lattices each consisting of different logical operations (AND, OR etc.). The code (written in VHDL) is translated into Boolean algebra which is used to make or break interconnections between the aforementioned lattices. The result is that logical expressions are built up that reflect the original code but are now realized in hardware.

Fixnetix has consulted with the SEC, ESMA, FSA and the Bank of England to provide an FPGA risk solution which protects at all levels; broker, trading participant and the market venue. In addition to the regulatory requirements that have been implemented since the Flash Crash of May 2010, many organizations seek to implement their own comprehensive compliance checks in order minimize their own risk profiles. In conjunction with the risk control capability is the client experience, which Fixnetix attempts to maintain through minimal order flow interference at all times. FPGAs are often thought of as derisory, due to ungraceful or unexpected operation when compared to traditional EMS or those usually operating under sponsored access directly with an exchange. Therefore, the Fixnetix approach of attempting to mirror the exchange behaviours has won favour with the largest hedge funds and proprietary trading firms in the world. This allows brokers offering the system, the ability to screen orders in microseconds with highly deterministic latency regardless of load, with the same or more comprehensive risk checks than in their current software EMS. Laudable capabilities include much reduced data centre requirements. The technology occupies a smaller footprint and power draw when compared to an equivalent server farm deployment that would be

The field part of the FPGA refers to the reprogrammability in the field, meaning that a new series of interconnects can be applied while the FPGA is deployed in a customer environment.

required to check every single order, over thousands of securities and scaling to hundreds of millions of messages per day. This paper details some of the approaches Fixnetix has taken during the development of iX-eCute and provides the reader with a comparison of other solutions available. The aim is to provide grounding in the Fixnetix FPGA technology and highlight why its considered the most advanced EMS platform in existence today.

Fixnetix FPGA deployment model

A typical Fixnetix iX-eCute deployment consists of a number of FPGA cards deployed in chassis connected to a number of client and trading venues. Figure 1 depicts such an implementation.

Figure 1 A typical Fixnetix FPGA deployment showing the major components and the information flow between them. As can be seen the FPGA is the central component of the deployment but importantly, is only a small part of an overall deployment that includes network connectivity, clients, trading venues, supervisory control, system management and regulatory reporting. By design, iX-eCute mimics the expected exchange behaviour verbatim. This is so that when clients connect into iX-eCute, no changes are required to the submitting engine. This methodology carries the overhead of resends, sequence number reconciliation and exchange-specific code requirements. All must be implemented in, or connected to the FPGA itself. Furthermore, to ensure that the client-side and venue-side order flow is separated, independent 10G Ethernet connections and network stacks are maintained within the FPGA.

There are ancillary processes that reside in the host x86 server which maintains an exhaust from the FPGA for regulatory reporting, resend storage, management and monitoring information to Fixnetix and a command and control interface to other Fixnetix products, such as ix-eye, a proprietary real-time window into the FPGA.

Fixnetix FPGA architecture

The internal architecture of iX-eCute is highly complex and is spread over millions of gates forming counters, adders, memory blocks, state machines and I/O interfaces. A key differentiator is that the Fixnetix FPGA is written directly in VHDL rather than using a higher level cross-compiler. This allows for much greater control of the resulting logic and a reduced latency design. A topological interpretation of the internal architecture of iX-eCute is described in Figure 2.

Figure 2 - Image of the components that make up the FPGA Network Stack The network stack is what allows the FPGA to communicate with order submission engines and trading venues over standard networking connections such as 10G Ethernet. It incorporates Ethernet framing and Checksum, IP routing and CRC and a full TCP stack. The stack includes standard features such as TCP ACK/Sequence number tracking, lost packet retransmissions, fast retransmit, Jumbo frames and VLAN tagging with support for 16/32/64 sessions. The stack is based on an industry leading and commercially available TCP engine that has been heavily customised to specifically target financial market applications, including protocol aware per session application message capabilities. Session management TCP allows for virtual independent connections or sessions between the server and host, which are interleaved serially into a single Ethernet connection. Hence, while the client may perceive a single and isolated connection, the reality is that the bandwidth is shared with other users. TCP will send each session data without consideration of application messages, so called packet fragmentation. This

means that orders from client A (and session A) and orders from client B (and session B) will not always arrive in the expect order. For example, (A) may be split between the orders from (B). Hence, the FPGA needs to perform message integrity checking not just at TCP/IP level but at the application message level as well; necessitating dedicated and protocol aware network components within the FPGA to ensure complete messages are forwarded to downstream logic. Protocol Parser As part of the core iX-eCute functionality, the application message must be interpreted and the information relevant to risk profiling extracted. This involves a process known as parsing, whereby preset triggers are searched for in the message and subsequently converted into a Fixnetix standard format. This may involve ASCII to binary price conversion, normalization of quantity or price values and translation of stock code/symbol. Each parser is unique and written to encapsulate the entire standard on which it is based, allowing any value or tag to be retrieved and converted on demand. Protocol Engine Operations that are required to communicate to a particular client or exchange but not associated with order flow are handled by the protocol engine. Heartbeats, login authentication, resend and sequence numbers are managed per client/session as dictated by the protocol. This area of the code is written as a direct implementation of the market specification, to allow iX-eCute to mirror the exact operation and behaviour of the market. This includes required tag validation, market specific behaviours to resends and Cancel-On-Disconnect (COD) and, where applicable, protocol translation. The latter often involves significant effort to map iX-eCute risk-related requirements and maintain conformity to given protocol. Business Logic The area is where the client, market or exchange specific risk operations are situated. Requirements, decoded into development blocks from initial specification documentation, are coded so as to perform risk, compliance and message manipulation (if required) within the FPGA. As this is a dedicated and separate module, it is possible to add functionality without any architectural or protocol changes which reduces risk and development time.

WHAT IS TCP?
There is often some confusion over what TCP really means. TCP is a method of guaranteeing the delivery of data broken into small chunks or packets over a network. It is commonly paired with IP (TCP/IP) which enables a route over a network for TCP traffic. The overheads of TCP when implemented in software are onerous and include acknowledgement of packets (ACK), the short term storage of each packet (Re-transmit buffers), multi-session managements and maintenance of sequence numbers.

Hence, there is a considerable advantage if TCP is encapsulated in an FPGA when compared to a software stack. Taking concepts from a pure software environment into hardware offers a significant challenge which is often why UDP or other lighter protocols are utilized.

Latency There is a great deal of confusion and misinformation regarding latency and how it is measured, specifically when financial trading systems are considered. For transparency, Fixnetix measures latency using a third party measurement device wire to wire. A small amount of the light used to carry signals over 10G Ethernet is optically tapped. Similarly, the same process is applied to the outgoing 10G Ethernet light signal. This method guarantees both that the signal is truly what is on the wire and that no latency is being added/subtracted by the measurement apparatus. The tapped light signal is fed into a device capable of sampling data every nanosecond. Thus, the output of the sampling device is a faithful representation of data flow at either end of the FPGA system. The latency quoted by Fixnetix includes latency from the 10G SFP+, the Ethernet/IP/TCP stack and Fixnetix IP, and risk checking processes. The number is initiated when the first byte of the packet arrives at the card until the last byte of the message leaves the card, so called first byte in last byte out. Often it is tempting to produce headline marketing figures stating greatly reduced times for a single operation within the FPGA but, in reality, clients will never see such times as not all the components of the system required for correct operation are included. It is also commonplace to produce latency statistics based on ideal conditions or as a best case (specifically when a software system is considered which will not provide deterministic latency). Fixnetix values are provided in conjunction with the type and length of message being measured, as detailed in Figure 3.

Figure 3 Fixnetix latency measurement method

What makes iX-eCute superior? One of the compelling features of iX-eCute is that is interprets and responds in the native protocol of the exchange. Importantly while others essentially parse the connection to the exchange operating much like a switch or firewall, iX-eCute terminates both the client and exchange sessions in the FPGA. This allows for graceful rejecting of orders via a reject message formatted in the same way as the exchange. Other less capable FPGA-based risk systems respond to risk breaches by either a sudden and unexpected disconnection of the network connection between the user and the FPGA, or by injecting erroneous information into the outgoing message to the exchange to force a rejection; the latter is a practice some exchanges are actively looking to penalize against. Another unique feature is the order life cycle management that is maintained within the FPGA. This allows for risk profiling on orders that could otherwise not be analysed. For example, OUCH amends orders for NASDAQ do not contain the original instrument. Therefore, if a calculation on the price % deviation is required the original order must be found to ascertain the original instrument, before a price lookup can commence. Another example is daily consideration, where the previous order chain is required to calculate the current position before the order is released to the market. The FPGA has a finite amount of memory available to it and therefore, sophisticated methods are required to manage the storage of salient information. As the order ID associated with the order is larger than the available memory space, a process known as hashing is used to provide a key into the memory. Fixnetix have a patent pending approach to this which results in no latency penalty for subsequent hash collisions until the memory is >75% full. In addition, the Fixnetix approach integrates client and venue resends and the management of sequence numbers on both side; the FPGA sources the data using its own session store separate from the exchange. The important distinction is that all of these features are required for the product to be full pre trade. Other solutions that do not offer such support are not fully implementing the client, exchange and marketplace requirements for risk control. iX-eCute offers a unique set of functions specifically to appear transparent to the trading participants, while offering a full set of regulatory and compliance checks at the lowest possible latency. Table 1 summarizes the main differences between iX-eCute and packet sniffing-type FPGA systems in use today.

HASHING?
Hashing is a method used to map alphanumeric strings into numerical addresses that map to an area of memory, addressable by the FPGA. In general this is a two stage approach. The first is the hashing (usually a CRC32) followed by truncation of the resulting hash to match the available memory size. Due to the truncation process, IDs that were originally unique will hash to the same location, defined as a hash collision. The remedial action usually undertaken is to search up or down the memory to find a spare location, so called linear searching.

There is an increasing latency penalty to linear searching and therefore, Fixnetix have devised a patented method of collision management without incurring any latency penalty within given memory constraints; Thus providing true deterministic latency.

Feature

iX-eCute

Packet sniffer
No Some No Partial No No <1uS No Some No No No Zero

Software EMS
Yes Yes Yes Yes Yes Yes 200uS Yes Yes Yes Yes Yes Large

Exchange-like order rejects Drop copies for back office Order life cycle storage Intraday Limit setting Independent TCP for a true order block Resend support System latency Custom Tag Support Multi-Client Support Single to Multi Session management FIX to Native Support Pre-trade Checks Jitter

Yes Yes Yes Yes Yes Yes 1-2.5uS Yes Yes Yes Yes Yes TCPDependant

Table 1 Fixnetix iX-eCute versus packet sniffing FPGA systems It can be seen that iX-eCute offers functionality similar to a traditional software EMS, but with a significantly reduced latency and robust deterministic performance. Packet-sniffing systems do not have many of the capabilities of either a software EMS or iX-eCute as they do not interpret and understand the application message. Therefore, they are not capable of performing a similar level of function to iX-eCute and do not offer a significant improvement in performance; they are ostensibly risk accumulators with a pre-set trigger. When the trigger is breached they are not capable of sending reject messages to the client and instead use more destructive methods, such as bringing down the physical connection between the client and the sniffer or, in some cases, polluting the packet and passing it to the exchange for rejection. The latter process is under scrutiny by some exchanges with a view to prohibiting such methods, as it is viewed as unnecessary message load on the exchange gateway.

Conclusion This paper provides an insight into how Fixnetix has achieved success in the FPGA trading space using a differentiating methodology of implementing all the components that are usually found in software EMS systems within an FPGA. This approach is not without complexity and requires innovative solutions to issues specific to FPGAs, such as memory management and session/packet fragmentation. With iX-eCute now live in 3 regions across the world and managing billions of dollars daily, it is evident that the methodology adopted for iX-eCute is highly regarded within the industry and offers a feature and performance combination that is second to none. Fixnetix have built on the foundations detailed here and two future papers are planned. The first will detail the innovations in simulation and system modelling, which enable Fixnetix to deliver on client expectations in terms of bug fixing and issue resolution turn-around that are comparable to traditional software development. The second will detail how Fixnetix has utilized its FPGA know-how to build a Fix to Native translation engine for the Tokyo Stock Exchange that encompasses persistence, enrichment, single to multi session management and advanced risk checking such as position and borrow lists.

Contact:

sales@fixnetix.com Fixnetix 9th Floor 33 King William Street London EC4R 9AS