Bill Roth, Ulitzer Editor-at-Large

Bill Roth

Subscribe to Bill Roth: eMailAlertsEmail Alerts
Get Bill Roth via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Blog Feed Post

LogLogic’s Universal Collection Framework

By Christophe Briguet

There’s been some back and forth on the interwebs about why we’re introducing the Universal Collection Framework, and why we intend to essentially give the underlying magic (the ULDP protocol) away to the competition. Most of the negative feedback (exclusively from the competition) has been misinformed. I’m not throwing stones here, we simply haven’t told them (or you) what UCF is, why we invented it, and why the industry needs it. Those we have told (analysts, press, etc) all seem to think we’re doing a good thing.

So let’s lift the covers and look at UCF in a little detail.

Why Invent Something New?

Large enterprises often have multiple smaller, remote facilities. Retail companies can have thousands of stores; the military can have thousands of deployments; financial institutions can have thousands of smaller facilities. A highly distributed network is now the norm, and these organizations all need to centralize their IT infrastructure data reliably and efficiently.

The standard logging protocol today is syslog, a tool designed for the LAN environment. It runs over UDP or TCP, and in our opinion, does not comply with WAN audit quality requirements: it does not guarantee the confidentiality and a reliable storage of the data it transports. Furthermore, there is no standard way of managing slow, loss-y, insecure, and potentially overloaded WANs. Streaming syslog over unreliable and unsecure connections is not acceptable in our opinion.

What is UCF?

The Universal Collection Framework (UCF) is LogLogic’s answer to the imprecision of data collection over unpredictable pipes. Built on the innovative Universal Loseless Data Protocol (ULDP), it provides WAN support for IT infrastructure data, including logs, SNMP, network flow, etc. The UCF consists of the Universal Collector software (UC), that can run in both agent and collector mode on various operating systems, and the Universal Collection Manager (UCM) - available as a virtual appliance.

UCF Features

The UCF provides the following:

A reliable two-phase commit.

  • Instead of simply sending the data (syslog UDP) or waiting for acknowledgement of receipt (syslog TCP), the UC waits for an acknowledgement that data has been committed to stable storage. The protocol allows for a delayed acknowledgement of data, so that the transfer of data can be pipelined (critical for WANs which can have long response times). 

Network smoothing.

  • Instead of unreliably sending the data (syslog UDP), creating backpressure on the log source (syslog TCP), or logging the data to a file for scheduled transfer (FTP), the UC delivers a smoothed stream of live data. When possible, the UC will send a real-time stream of data. When the data stream exceeds the available network bandwidth (including a network outage), the UC sends as much data as possible, while buffering the remainder. When the bandwidth returns, the UC will transmit the data at a higher than normal rate.

Versioned protocol.

  • ULDP supports a variety of protocol options (compression, encryption, scheduled transfers) and will likely support more in the future. In a distributed environment, however, customers cannot afford the complexity of trying to keep all their software synchronized. The ULDP protocol is fully versioned, enabling negotiation between any two components for the features that it chooses to enable. As a result, customers can leverage the greatest common factor of options between components, rather than the least common denominator, and upgrade any component of the infrastructure as necessary, when practicable.

Smart packeting

  • The UCF supports over-the-wire compression and encryption (SSL) of the data. This reduces the total amount of bandwidth consumed and eliminates the need to have a VPN between UC and UC recipient. 

Remote management

  • The UCM drastically reduces the overhead of ongoing management. Once a UC is installed, the administrator almost never has to physically touch it again - all changes can be done remotely, including configuration changes (adding a new log sources, changing the filtering policy), or updating the software (migrate from v2 to v3 etc.). The UCM also provides increased management visibility of the log collection layer by providing statistical information and analysis.

Be believe UCF is unique. Nobody else offers a secure WAN aware way of moving IT data from remote locations on a schedule that suits your business. We intend to open source the underlying protocol because we think the industry needs it. We recognize that at that point we’ll loose control over our invention, and others may be able to build a better UCF. But that’s okay, the industry needs this, syslog is simply not up to the task.

Read the original blog entry...

More Stories By Bill Roth

Bill Roth is a Silicon Valley veteran with over 20 years in the industry. He has played numerous product marketing, product management and engineering roles at companies like BEA, Sun, Morgan Stanley, and EBay Enterprise. He was recently named one of the World's 30 Most Influential Cloud Bloggers.