The Teraflow Testbed (TFT) is an international application network for exploring, integrating, analyzing, and detecting changes in massive and distributed data over wide area high performance networks. Today, the Teraflow Testbed analyzes 1 and 10 Gpbs streaming data. The goal of the Teraflow Testbed is to develop innovative technology for data streams at these speeds and to prototype technology that can be deployed over the next decade to analyze 100 Gbps and 1000 Gbps streams.

The TFT consists of computer clusters distributed over three continents that can transmit, process, and mine very high volume data flows, or what we call teraflows.

The clusters are integrated using advanced 10 Gbps photonic networks and rely on both Layer 2 optical switching and Layer 3 routers. There are TFT clusters in Chicago, Illinois; Kingston, Ontario; Amsterdam, The Netherlands; Geneva, Switerland; and Tokyo, Japan. It is anticipated that additional clusters will be added in the near future.

Using the TFT, NCDM and its partners are developing: 1) new network protocols that can exploit the bandwidth available on these networks; 2) high performance web services; and, 3) data integration and machine learning services that scale to teraflows. Up to now, distributed applications have generally been designed to minimize the movement of data and instead move the queries and the computations; with teraflow services, a new class of applications can be developed that also move the data when required so that all relevant data can be analyzed.

The TFT middleware is designed for both 1 Gbps and 10 Gbps routed networks as well as new emerging networks in which applications can set up, tear down, and monitor their own optical paths. The TFT middleware is open source and includes the UDT protocol for high performance data transport; the SOAP* high performance web services; and high performance Preditive Model Markup Language (PMML) based scoring engines.

The Teraflow Testbed is sponsored in part by the National Science Foundation, the US Army, the Department of Energy, and the University of Illinois at Chicago.