comparing solace’s broker- based messaging … · comparing solace’s broker-based messaging...
TRANSCRIPT
Copyright © Solace Systems, Inc.
http://www.solacesystems.com
Comparing Solace’s Broker-Based Messaging Architecture with Peer-to-Peer Architecture
When considering messaging middleware technologies, it’s important to understand the business
requirements of a given application and consider them in context of the characteristics and
strengths of the technology’s underlying architecture.
This paper compares peer-to-peer messaging architecture with broker based architecture, and in
particular Solace’s hardware-based message router. It explains why peer-to-peer architecture is
inappropriate for the vast majority of scenarios because it sacrifices mission-critical characteristics
capabilities in the attempt to reduce latency well beyond the point that it matters.
Table of Contents 1 Introduction .......................................................................................... 2
2 Architecture Descriptions ...................................................................... 3
2.1 Peer-to-Peer Architecture ..................................................................................................... 3
2.2 Solace Hardware-Based Broker Architecture ........................................................................ 5
3 Critical Messaging Characteristics ........................................................ 7
3.1 System Simplicity ................................................................................................................. 7
3.2 Policy Control, Status Monitoring, Capacity Planning, Troubleshooting ................................ 9
3.3 Security ................................................................................................................................ 10
3.4 System Scalability (while maintaining simplicity) ................................................................... 12
3.5 System Robustness, Predictability and Real-Time Decoupling ............................................. 10
3.6 Enterprise grade, robust and feature rich persistent queuing ................................................ 14
3.7 Lower application development costs and fast time to market .............................................. 15
4 Summary .............................................................................................. 15
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
2
1 Introduction The peer-to-peer architecture optimizes one particular characteristic, namely latency, to a
degree that is not important to most applications1. This complete focus on latency leads to
sacrificing other characteristics that are much more important to the vast majority of
applications. This includes financial services applications such as: order management
systems (OMS), middle office, global reference data distribution, many market data
distribution use cases, single dealer platforms, global trade and risk aggregation.
Outside of finance, this includes applications such as ESB, SOA and EDA backbones,
eCommerce, betting and gaming platforms, real-time control platforms, and automated
processing systems. All of these applications value performance to some degree, but other
features are more important than achieving sub 10 microseconds of latency.
Achieving that level of performance is important for algorithmic and high-frequency trading
applications, however this paper is not intended to include the ultra-low latency scenarios
where it’s critical to shave off every possible microsecond because stakes in algo/HFT
trading require different assumptions and compromises.
This paper is focused on the overwhelming majority of applications that value and can’t
sacrifice the following characteristics:
1. Simplicity
2. Policy control, status monitoring, capacity planning and troubleshooting
3. System robustness, predictability and real-time decoupling
4. Security
5. Scalability (while maintaining simplicity)
6. Enterprise grade, robust and feature rich persistent queuing
7. Lower application development costs and fast time to market
Each of these points will be discussed in detail, but first we provide a description of both the
typical peer-to-peer architecture and the Solace hardware-based broker architecture as
necessary background to the discussion.
1 Some claim that providing large fanout of data using multicast is an advantage of peer-to-peer over broker-based architectures but broker-
based solutions can also support multicast delivery. In fact, when brokers provide multicast delivery, they isolate publishers from retransmit
requests (NACKs) from misbehaving or slow consumers.
Solace’s hardware-
based message
broker meets many
key needs that trump
the theoretical
advantages of peer to
peer systems, many
of which don’t hold up
in real-world
implementations
anyway.
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
3
2 Architecture Descriptions
2.1 Peer-to-Peer
Architecture While there will be variations among
products, the basic peer-to-peer
“nothing in the middle” architecture is
as depicted below. The figure shows
servers housing publisher/subscriber
applications on the bottom and
servers housing messaging
infrastructure functionality at the very
top. Connectivity between these
servers is via a multicast network,
which means all communicating
parties must be on the same layer 2
subnet, otherwise IP multicast routing
must be used between subnets,
which is not typical.
Since there’s “nothing in the middle”, any functionality other than direct application-to-
application non-persistent messaging needs to be implemented at the edge. Such typical
additional components are shown here as the orange server icons and in summary, these
functions include:
2.1.1 Store Processes
Store processes provide persistence by storing messages and their delivery state to a local
or remote disk. In most cases, store processes are deployed in odd numbers of three or
more and use quorum consensus or some other dynamic election mechanism to determine
whether the persistence service is operational.
Message delivery occurs directly from publisher to subscriber with the persistence
occurring in parallel to minimize latency. Each message and all subscriber ACKs must be
processed and stored by all (three) store processes. In the case of lost messages and slow
or offline subscribers, subscribers contact a store process to retrieve their queued
messages and rejoin the real-time delivery stream if and when they catch up.
In the case of a store process failing or becoming too slow, synchronization with other store
processes comes into play. Such synchronization is a complex process that products
handle differently, if at all. Another function that products handle differently is if and how
they replicate and activate configuration changes to each store process.
It’s frequently claimed that since peer-to-peer message delivery does not require the store
process to filter and deliver messages to subscribers, store processes offer better
performance than brokers. While this may be true when all subscribers are online and
behaving well, this architecture degrades to a “store and forward” broker architecture for all
subscribers who fall far enough behind the real-time message flow that they need to
recover from the store process and thus this performance advantage is negated.
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
4
2.1.2 Inter-subnet Gateways
Since peer-to-peer solutions count on multicast to effect
the one-to-many message replication, the “nothing in the
middle” advantage only applies to applications where the
publisher and subscriber are on the same IP subnet. To
reach applications in other subnets or other datacenters,
TCP gateways must be deployed at the edge which
receive the multicast traffic, filter by topic and then forward
to one or more remote gateways over TCP – like a broker
would do. These gateways need to be deployed in a
redundant manner for high availability, and must be
continuously monitored.
2.1.3 Desktop Distribution Gateways
Similar to inter-subnet gateways, gateways are required to distribute data to desktops over
TCP since multicast is confined to the local server subnet
2.2 High Availability and use of TCP Each of the additional components described above needs to be deployed in a redundant
manner on a separate server in order to provide high availability. From an operational point
of view, each component must be monitored, configured, managed and upgraded.
Even though most applications don’t send each message to many subscribers (aka “high
fanout”), multicast is frequently used to enable communications between applications. This
forces networking teams to manage multicast traffic and makes application teams deal with
the allocation and engineering of multicast groups. To not use multicast would require an
N**2 mesh of TCP connections between applications as shown here, which introduces the
issues of TCP connection scaling, topic resolution maps, message replication burden on
publishers—to name a few.
2.3 Multi-site Architectures As messaging is typically required in multiple subnets
or in multiple datacenters, this infrastructure must be
replicated and interconnected with gateways as shown
here. So the only situation in which peer-to-peer
solutions are advantageous is when the need for
latency under 10 microseconds is mission critical and
the most important factor.
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
5
2.4 Solace Hardware-Based Broker Architecture Solace’s purpose-built message router provides the important characteristics and
functionality of a broker without the performance penalty introduced by software-based
brokers.
Solace can route millions of non-persistent messages per second and hundreds of
thousands of persistent messages per second. Due to the broker-based architecture
employed, these high rates can be provided with low latency, and without sacrificing the
many other key technical characteristics most applications require.
This diagram shows the same
application deployment as above
with a Solace broker-based
architecture.
Each application uses a software
API to connect to a Solace message
router via TCP—no multicast. The
message router filters, routes,
queues, replicates and retransmits
messages toward subscriber
applications freeing publishers and
subscribers from those
responsibilities – functions that are
performed within the messaging
APIs of these applications in a peer-
to-peer architecture.
The message router performs all the
messaging functions described in Section 2.1 but without any additional components.
Here’s how:
2.4.1 Persistence
Solace’s patented solution handles persistence with dedicated cards called Assured
Delivery Blades that are integrated into the message router. These ADBs enable failsafe
persistent message queuing at very high rate with low consistent latency. An external disk
array accessed via fibre channel or SAN is used as overflow to store messages when slow
or offline consumers are unable to receive them or keep up with the message flow.High
availability is provided via a second message router which maintains the same delivery
state as the active message router in RAM, and is ready to automatically take over in a few
seconds if the active message router fails. In the unlikely event of a datacenter-level failure,
all messages and their delivery state are flushed to non-volatile memory. To learn more go
to http://solacesystems.com/products/guaranteed-messaging
2.4.2 Inter-subnet messaging
Since TCP is used to communicate to applications, they are not required to be in the same
subnet as the Solace message router. They can connect to any message router across
subnets and even over the WAN. Communication to message routers in other datacenters
is performed by configuring TCP neighbor adjacencies between message routers. No
additional components are required and messages flow between the message routers
using Solace dynamic topic routing protocols.
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
6
Solace also supports datapath features that optimize the efficiency and performance of
communication over the WAN such as streaming compression, parallel TCP connections
and large TCP windows, so no additional inter-subnet gateways are required
2.4.3 Desktop Distribution
Similarly, messages can be fanned out to desktops using TCP to thick .NET or Java clients
without requiring additional fanout gateways or daemons.
2.4.4 Internet/Mobile Streaming
Communication to HTML5, Silverlight and Flex Rich Internet Applications (RIAs) as well as
to tablet and mobile devices over an HTTP infrastructure of proxies and firewalls is also
supported from the same message router without needing additional HTTP Push gateways.
2.4.5 Inter-Datacenter Connectivity
Supporting the same type of inter-datacenter connectivity as
described in Section 2.1 is shown in the following figure. No
additional gateways are required since the message routers
used to provide connectivity between applications within a
datacenter also provide the “gateway” functionality required
to link remote datacenters in a true subscription-based
pub/sub manner.
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
7
3 Critical Messaging Characteristics
3.1 System Simplicity Application architectures must be designed from the beginning to be as simple as possible
since they tend to become more complex as they evolve over time. High system complexity
translates to measurable expense in terms of higher operational costs, lower reliability
(more production outages), and increased time to repair production problems (thus longer
outage time per incident). Complex systems are also less flexible, which hampers the
deployment and modification of applications, which reduces business agility and slows time
to market.
Two things make the peer-to-peer architecture inherently complex: The number of
distributed messaging components, and the tight coupling among client applications and
between client applications and the underlying network.
Furthermore, the complexity of a peer-to-peer system increases more rapidly than that of a
Solace brokered architecture as the system scales and applications are added, amplifying
the operational cost, reliability and flexibility issues.
System simplicity can be described in terms of operational simplicity for the messaging
infrastructure and for the network infrastructure.
3.1.1 Operational simplicity of the messaging infrastructure
The peer-to-peer architecture with “nothing in the middle” results in a messaging
infrastructure composed of loosely coupled, distributed components for different functions
deployed “at the edge”.
o Complexity of Components: Since each functional component (such as store
processes, desktop and inter-subnet gateways and web/mobile gateways) must
provide high availability and scale for capacity, many instances of each on separate
servers must be deployed to ensure that a given function can survive any single
fault. This requires management of more servers, OS and networking in addition to
the messaging components.
o Capacity Planning: If you combine different functions on the same server for
consolidation (e.g. a store process and a gateway process), then the infrastructure
team needs to ensure that there are sufficient resources for both functions to
operate properly and to monitor that over time.
o Troubleshooting: Identifying and addressing system faults is very difficult in such a
distributed environment where each component has multiple instances to
accomplish a single function.
o Redundancy Across Subnets: Since peer-to-peer draws heavily upon multicast,
which is typically confined to a subnet, this same infrastructure of gateways and
stores must be repeated in each subnet where server applications reside.
o Duplication for Separate Applications: Providing messaging services to many
independent application groups who must be isolated from each other from a
performance and topic namespace point of view requires duplication of the above
infrastructure.
The complexity of a
peer-to-peer system
increases more quickly
than that of a brokered
architecture as the
system scales and
applications are added,
amplifying the
operational cost,
reliability and flexibility
issues.
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
8
The end result with peer-to-peer is lots of moving software components on many servers in
each subnet which means lots of distributed state and status to deal with often without the
benefit of a centralized management function which can correlate and roll up overall status
and coordinate functions such as configuration changes and software upgrades/rollbacks.
In contrast, the Solace architecture provides:
o Integrated Solution: All functional components are integrated into a message
router instead of being distributed throughout the datacenter. There are no separate
persistent store or gateway processes and servers.
o Unified Platform: Connectivity over the LAN, WAN or Web to browsers and mobiles
is provided by the same infrastructure from a single device.
o Integrated HA: High availability is provided by redundant components within the
message router and at the device level for all functions. These HA functions use
hard-wired communications between message routers (e.g. fiberoptic links on the
Assured Delivery Blade) and networking standards proven by the IETF for maximum
reliability and simplicity
o Shareable Platform: The high performance of the Solace message router
combined with built-in virtualization allows a single message router to be shared by
multiple applications, but each with its own private message broker, thereby
reducing datacenter sprawl and the number of managed entities.
o Integrated Disaster Recovery: Persistent message replication to a disaster
recovery site for business continuity is performed directly between the message
routers without the need for additional equipment and without needing to coordinate
use of storage replication. Both synchronous and asynchronous replication are
supported.
3.1.2 Operational simplicity for networking
Peer-to-peer architectures rely on Ethernet multicast for replication rather than a full mesh
of TCP connections among all clients, and each publisher must perform its own message
routing and replication. Multicast may be useful for the peer-to-peer messaging
infrastructure itself so that the many Store and Gateway processes receive each message,
but it is not appropriate for the vast majority of applications since typically only a very small
proportion of subscribers receive any particular message. Still, all clients who join a
multicast group must receive and filter all messages on that multicast group.
In addition, multicast brings with it well-known problems such as multicast group
management and engineering, network congestion and discards due to lack of flow control
and non-uniform link speeds, retransmission requirements and tuning, crying baby
syndrome and so on.
In the Solace architecture, all clients connect to the message router using TCP, avoiding all
the pitfalls of multicast. Network congestion, link speed mismatches, slow consumers and
isolation of misbehaving clients are all gracefully handled by the message router to protect
the network and clients from overload and collapse. The message router performs per-
client message filtering based on subscriptions and only sends clients messages they have
subscribed to at a rate they can handle.
Clients connect to
Solace message
routers using TCP,
avoiding all the
pitfalls of multicast.
The message router
handles network
congestion, link
speed mismatches,
slow consumers and
misbehaving clients.
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
9
3.2 Policy Control, Status Monitoring, Capacity
Planning and Troubleshooting Being able to monitor, capacity plan, troubleshoot and control a messaging environment is
a critical requirement of any production system. Achieving operational excellence hinges on
the ability to proactively plan to avoid outages and quickly resolve production problems
when they do occur.
In a peer-to-peer architecture messages don’t flow through a messaging-aware
intermediary, which makes it impossible to do things like:
o Monitor per-client queue depths, packet loss to/from a client, determine which clients
are slow (and why), which publishers are publishing too fast, which subscribers are
subscribing to what topics
o Capacity plan by looking at per client send/receive message rates – both
instantaneous and averaged
o Enforce per-client policy controls such as rate limiting, resource limitations and
security
o Know which applications are up and connected
Without a broker, administrators must either snoop and decode multicast traffic using a
span port to pinpoint the problem, or instrument all applications to respond to management
requests sent from some management station. Managing via protocol decodes requires
extremely specialized and technical staff and is not feasible for all but the most
sophisticated organizations.
Instrumenting applications to aid management is expensive and fundamentally flawed
because it assumes that the network, servers and applications are all“sane” enough to
provide valid information, but that very lack of sanity is usually the problem that needs to be
isolated and resolved.
If an application is hung or in a loop or CPU is at 100% or there is a networking problem
affecting the server, then there is no way to gather status. Even if you did determine the
cause of the problem and wanted to take action (such as disconnecting an application from
the messaging system) you’d need to build your own method for doing so because even if
there were mechanisms built into the messaging system, they wouldn’t work in cases
where there are networking, server or application problems.
In the Solace architecture, all inter-application traffic flows through the Solace message
router so it becomes the “central nervous system” of application monitoring. The message
router provides per-client visibility at all layers so administrators can see packet retransmits
in each direction, along with instantaneous queuing levels and high water marks, in/out
message rates, connection setup attempts, subscription lists, round-trip-time latency, and a
listing of fastest publishers/subscribers to name but a few.
Solace message routers can enforce per-client policies such as message rate limiting and
can also restrict or limit access to various resources. If you have a rogue application that is
misbehaving, administrators can pinpoint and disconnect it.
There is no need for the application, its server or the network to be sane to detect the
problem and take corrective action. Management of the Solace message router is via a
All inter-application
traffic flows through
the Solace message
router so it becomes
the “central nervous
system” of application
monitoring, with per-
client visibility.
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
10
dedicated management network port to support a separate “command and control”
network. Since the control plane of the message router is separate from the hardware
datapath, the message router is always responsive to management requests and can
always gather detailed status information regardless of how much messaging traffic is
being routed. This type of detailed and reliable management is simply not possible in a
peer-to-peer architecture.
3.3 System Robustness, Predictability
and Real-Time Decoupling System robustness is the ability of a system as a whole to function in the face of
component-level failures. This means containing the impact of faults so they don’t spread
throughout the system, and being able to quickly find and fix faults.
In messaging systems, typical faults include misbehaving consumers such as slow or
crashing consumers or consumers who cannot keep up with bursts, networking faults (at a
server or within the network) and intermittent network congestion.
In the peer-to-peer architecture, the lack of an intermediary or “shock absorber” creates a
hard wired real-time coupling both between client applications and between clients and the
network which makes the overall system fragile and prone to collapse. This coupling exists
because when a fault occurs, the peers in the system must work together to overcome it
since there is no intermediary to do so.
In the peer-to-peer architecture, all consumers and the network must keep up with both the
packet and (unfiltered) message rates from all multicast groups they have joined whether
they are messages they want or not. Packet loss due to a slow subscriber requires NACK
processing and retransmission directly from the publisher, thereby increasing the load on
the publisher. In some products, retransmissions are multicast to many subscribers – so a
single misbehaving client can affect other properly operating clients. During packet
recovery, applications must buffer all newly arriving packets until the lost one is received.
This places additional memory and processing burden on an already taxed subscriber
which can cause it to drop more packets thereby affecting more publishers – hence the
potential for collapse.
Packet loss in the network, typically many packets in bursts, is even more problematic as it
impacts all downstream consumers causing them all to send NACKs to all publishers of
dropped packets requesting retransmissions, thereby further increasing demands on all
affected publishers. These situations become even more likely as systems scale with more
endpoints, networks become more complex and as networks and servers become less
homogeneous, thereby giving rise to speed mismatches that cause packet loss.
With peer-to-peer, there is no way to flow control the publisher (as it would delay traffic to
all other consumers) nor can you isolate applications because there is nothing to provide
the isolation. Publishers publish as fast as they can and generate more traffic to gap fill for
packet loss, whether this traffic is helping or not. Many messaging systems provide
configurable parameters for these behaviours, but knowing how to tune them for all
conditions can be difficult to avoid the “fix here, break there” situation. It’s also difficult to
determine where and why packet loss is occurring and where the traffic is coming from due
to lack of centralized management. Even worse, such storms make the network unreliable
so it can be even more difficult to troubleshoot.
In peer-to-peer
systems the lack of
an intermediary
creates a hard-wired
real-time coupling
between publishers
and subscribers,
which makes the
system fragile.
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
11
All the above mechanisms lead to a system that has good performance in a small, uniform,
highly managed environment, but performance and stability quickly deteriorate as rates
increase, the system scales, the communication patterns become more chaotic, the
environment gets less uniform or less high touch managed.
Especially in cases where most messages go to only a small number of subscribers, the
problems associated with multicast far exceed their benefits.
In the Solace architecture, message delivery is tailored to the needs and abilities of each
client. The only messages sent to subscribers are for topics that they have subscribed to –
not all messages in a multicast group. This means that:
o An application only needs to scale with the messages it needs to process – not at
the rate someone decided for a multicast group.
o The bandwidth to servers and CPU processing is reduced since message filtering is
performed by the message router.
o Slow consumer handling is more orderly and controlled with no impact to other
clients in the system
This leads to a system with higher performance that’s much more predictable as system
load increases and the system scales.
The Solace message router provides an orderly, controlled and isolated handling of
misbehaving consumers as follows:
o When the offered load is too high for a subscriber for a period of time, the TCP
window from the message router to the consumer closes and no more messages
are sent. Neither the network nor the client is bombarded with traffic they can’t
handle.
o Messages for the slow client are queued by the message router to guard against
message loss. Queue lengths are configurable and additional features such as rate
controls and message overwriting (called “eliding”) can be applied by the message
router to help that consumer.
o In addition to message pre-filtering, the message router improves the efficiency of
slow client processing by bundling many small messages into fewer larger packets
for that client, reducing per-packet processing overhead.
o In the case of packet loss in the network, the message router interacts with the client
to gap fill so there is never any impact on the publisher. Also, if N packets are lost
within the network only those N destination applications are affected – not all
applications that have joined the multicast group that the packet was sent on.
Again, this contains the scope of impact.
o The fact that the message router is queuing and point-to-point delivering messages
to an application also provides an infrastructure point to monitor message rates,
queuing levels and packet loss for each client and asynchronously notify the
operations team of anomalies without any application code being developed.
Rather than the unavoidable hardwired inter-client coupling effect necessitated by peer-to-
peer, Solace’s hardware brokered architecture provides a real-time shock absorber and a
true decoupling mechanism for inter-application communication. This in turn creates an
architecture that is more forgiving to faults in the system and provides the isolation required
With Solace message
delivery is tailored to
the needs and
abilities of each client,
and messages are
only sent to
applications that are
subscribed to their
specific topic.
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
12
to contain the fault and avoid systemic collapse while at the same time providing critical
management visibility as to the location and cause of the fault.
3.4 Security The security requirements of applications vary greatly depending on their regulatory
environment and organizational policies, but most mission critical deployments require one
or many of the following:
o Authentication: Restrict access so not just any application or user can connect to
the message-level communication bus. Authentication can be as simple as
username/password or more complex requiring X.509 certificates or one-time
passwords.
o Authorization: Maintain entitlement control over which applications can publish and
receive certain content.
o Encryption: Ensure that no data between applications is in the clear on the
network.
Such precautions could be required to prevent internal deployment mistakes or guard
against malicious or illegal actions. In many applications, the lack of one or more of these
capabilities renders that technology ineligible for use.
Peer-to-peer architectures can’t meet these requirements due to the lack of a messaging
intermediary to enforce them. Network level mechanisms such as IEEE 802.1X can be
used to provide server-level authenticated access to the network, but they do not provide
application-level authentication, nor do they support multicast group access controls, much
less enforce per message topic-based publish/subscribe controls. Similarly, transport level
encryption standards such as SSL and TLS are not supported by multicast. You can’t
avoid the issues of multicast by moving to a full mesh of TCP connections as a transport
between applications because that would require all peers to support server-side functions
such as authentication, authorization, certificate management, etc..
In the Solace broker-based architecture, client applications must connect via TCP to the
Solace message router to access the message bus in a client-server model. Per-client
authentication is supported, per-client topic-based publish/subscribe access control rules
can be enforced in real-time on a per-message basis, and transport level security of various
strengths is supported using SSL/TLS.
3.5 System Scalability (while maintaining simplicity) Messaging deployments typically scale in many dimensions over time –number of client
endpoints, message rates, number of separate applications, number of sites, number of
subnets, etc. The ability to scale each dimension in a manner that is cost effective and
minimizes operational complexity is an important consideration when selecting a
messaging infrastructure.
In the peer-to-peer architecture, the only way to scale the system is to add more multicast
groups, gateways, store processes and requisite servers.
In general, more hardware and software infrastructure components are added at the edges
to accommodate scale (and yet more for high availability) for each function of the
messaging system. For example, to add a second application requiring persistence that
P2P messaging
systems can’t
implement
authentication,
authorization or
encryption because
there is no
intermediary to
enforce them.
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
13
requires isolation from the first application, typically 3 more store processes need to be
deployed on 3 additional servers.
Other peer-to-peer considerations as a system scales include:
o Messaging mechanisms that were unnoticed in small deployments become
problematic. Examples include topic resolution cache sizes, retransmission cache
sizes, topic to ID binding refresh rates and aging, retransmission parameters, etc.
Not only do you need to monitor and tune these parameters, you need to deploy
new settings to all client applications which can be a management nightmare.
o Maintaining a uniform hardware environment is also more difficult and expensive as
the system scales, and not doing so leads to a fragile system and the need for
messaging tuning. For example, a heterogeneous 10GE/1GE network is more likely
to cause packet loss at the 10:1 link speed reduction points. Such network-based
based packet loss affects all downstream consumers and is more likely with UDP
traffic than with TCP traffic due to lack of flow control in UDP/multicast.
In the Solace architecture, scalability is achieved as follows:
o Virtualization: Each message router can support the addition of new independent
applications through the easy creation of“virtual message brokers.” This provides
significant savings and lets you offer messaging as a service in the network.
o Supporting clients in different subnets does not require additional gateways or
servers since those clients can connect to a common Solace message router due to
the use of TCP instead of multicast.
o Scaling to connect to other sites doesn’t require additional gateways or servers
either, since message routers can be configured for inter-device adjacencies
regardless of location.
o Each message router can support thousands of concurrent connections, and handle
throughput of 24M non-persistent msgs/sec and 450,000 persistent messages per
second. It’s easy to add more capacity with the installation of a new message router,
or pair of message routers for HA. On a per-virtual broker basis, these message
router pairs can be neighbored or federated to allow clients within the same
application to communicate with each other even though they are connected to
different message routers.
o Issues such as topic discovery and resolution mechanisms on each client do not
exist in the Solace architecture since the message router maintains this state and
brokers this communication, thereby decoupling publishers and subscribers from
each other and dramatically reducing the amount of client-based configuration
required.
o The network and server environments need not be as uniform as with peer-to-peer
multicast since the message router performs per-client filtering and uses unicast
TCP with its flow control to navigate heterogeneous networks in a controlled
manner.
In many dimensions, due to the hardware datapath and virtualization capabilities, scaling a
system based on Solace requires no additional investment, hardware deployment or
complexity and in cases where additional message routers are required, the inter-device
routing capabilities make federation operationally simple to achieve.
Thanks to high
capacity and
virtualization, it’s easy
to scale Solace-
based messaging
systems to support
new applications,
messaging
functionality or
capacity
requirements.
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
14
3.6 Enterprise-grade, robust and
feature rich persistent queuing Other than market data and sensor networks, the vast majority of distributed applications
require message persistence in their messaging infrastructure since message loss for any
reason is not acceptable. Many require high rate, many require sophisticated queuing
capabilities and some require low latency, but all of them require a fault tolerant, highly
available system that doesn’t lose messages – and this most often is far more important
than driving latency below 10 microseconds.
In the peer-to-peer architecture, persistence is handled using a technique known as
“parallel persistence” by store processes which are special subscribers to the multicast bus
as described in Section 2.1. The goal of parallel persistence is to minimize latency within a
small solution rather than addressing the needs of an enterprise messaging system
because:
o Parallel persistence only works within a subnet. Beyond that, if it is supported at all,
gateways are required between the subnets which then starts to look a lot like a
brokered solution, only it involves both store processes and gateway process
making it more complex than a broker solution.
o Ensuring storage integrity and synchronization with three independently running
store processes is fragile – especially in the face of store process outages and
networking problems
o Providing replication to remote datacenters for disaster recovery is an afterthought in
most peer-to-peer systems since the active site has three (or more) copies of the
data.
o Increasing the number of independent applications using persistence can require
deploying another set of store processes for each new application to provide
appropriate isolation, thereby creating much larger datacenter sprawl than with
software brokers
o Important messaging capabilities such as transactions, selectors, queue overflow
handling and sophisticated queue servicing algorithms that exist to facilitate
application architectures are completely at odds with parallel delivery, and in some
cases impossible to implement with parallel delivery. For example, you can’t deliver
messages in a transaction until the transaction is committed as it might be aborted
by the application.
While it is true that many peer-to-peer products also support the ability to have the store
process receive messages from publishers and forward to subscribers that makes it a
brokered architecture with all the associated software performance issues but without the
management of a full broker-based system.
The Solace persistence architecture uses patented techniques and custom-built hardware
specifically designed for the needs of high rate, low latency “never lose a message”
enterprise wide persistent queuing to overcome the bottlenecks associated with file
systems and rotating and or solid state storage systems. By providing such a high
performance broker, the Solace architecture provides all the benefits of a brokered
architecture without the performance and real-time drawbacks of a software
implementation:
By providing high
performance broker,
Solace offers the
benefits of a brokered
architecture without
the performance
limitations and other
drawbacks of
software brokers.
Comparing Solace’s Broker-Based Messaging Architecture with Peer to Peer Architecture
15
o Messages and message delivery state are synchronously stored on two message
routers and synchronized across message router activity switches at up to 450,000
msgs/sec.
o Messages can be synchronously or asynchronously replicated and queue status
synchronized to a remote site to provide disaster recovery functionality without
needing to involve storage replication such as SRDF.
o The store and forward queuing enables support for transactions, selectors, various
queue servicing disciplines.
o Message routers can be federated to allow producers and consumers of persistent
messages to be in different datacenters or in different continents.
o Management, monitoring and high availability are all built-in and since all messages
flow through the message router, message rates and queue levels can all be
monitored and asynchronous events generated when thresholds are exceeded for
maximum visibility.
3.7 Lower application development costs and fast
time to market Due to combinations of the many points above, application development, deployment and
operations costs are significantly less with a Solace hardware broker-based architecture
than with peer-to-peer. The principle reasons are fewer moving pieces in production, lower
complexity of the overall system, fewer parameters to tune, ubiquitous reach and integrated
platform for management and high availability. Solace delivers a completely integrated
messaging solution rather than the components of a solution that need to be integrated,
engineered and made manageable by the customer and then maintained on an ongoing
basis. In doing so, Solace customers can focus their energy on their differentiators and
what makes them great rather than building infrastructure.
4 Summary Messaging systems that have been deployed at an enterprise scale have historically had
some type of broker or daemon between client applications for many of the reasons listed
in this paper. Recent interest in peer-to-peer architectures for high performance
applications has been driven by the poor performance and predictability of software broker
messaging products, and led some to consider deploying such peer-to-peer products at
larger than departmental scale.
Solace’s hardware-based solution changes the game by providing all the benefits of a
brokered architecture with the high rate, low consistent latency of a hardware solution
purpose built to meet the needs of high performance, enterprise wide messaging.
To learn more visit
solacesystems.com or
call +1 613-271-1010.