Open letter to Minister of State for Public Service Reform, Mr. Brian Hayes TD
Hi Brian,
I would like to share a quick response to my reading of the Irish times article: State to demand price cuts from suppliers to reduce €16bn bill.
In order to best serve the needs of the Irish people right now and into the future, you need to seriously consider open source IT solutions. Across all departments and across all of Europe, government IT departments should be collectively investing in free open source solutions that solve their common IT needs.
Investment in open source IT solutions seeds innovation and is a commitment to shared future value. Investment in proprietary IT solutions is an innovation tax and a commitment to repetition.
This is not some sort of Marxist rant; open source is the best way to innovate. Open source software is a key reason amazon, google, twitter, facebook etc. emerged; they stand on the shoulders of giants.
I imagine this would mean a small shift in how government IT is organised. You would need to extract real value from the smart people therein. Rather than out sourcing decisions to global consultancy companies, you allow a shared need to be met from within.
You enable innovation, by allowing smart individuals to take ownership of both the problem and the solution and most importantly, to share the fruit of their labour.
The bottom line is this, all of the government departments have IT needs in common, they are much more alike than they wish to admit. The also share these needs with other governments thoroughout Europe.
There is no reason to constantly reinvent the wheel. We just need to enable people to share and evolve the best designs. Open source provides the freedom and motivation to do just that.
Apr 22, 2011
Apr 10, 2011
Consider Unhosted and open source for eHealth and eGov #DERIopenDay
DERI Galway produced an insightful open day on their developments in the semantic web of linked data. While I listened, two thoughts kept recurring that I want to explore. Chances are I am preaching to the choir but shucks, just in case I am not...
For a web architecture of the future look at Unhosted
At the root of the problem of siloed data and fragmentation (the database hugging phenomena) is the issue of ownership. Institutions have data that they don't really own because that data is of a personal nature. The collection of data is theirs, but not the individual components.
With Unhosted, the ownership problem is turned on its head. Users and aggregators of data only have a 'handle' (a URI) to personal data. A handle that is only useable with permission. Collections of data containing 'handles' can safely be shared. Granted, lots of issues need to be ironed out, but I think the architecture is on the right track and the concept is bang on.
Open source your research
Lots of what you do is plumbing. For new plumbing to be broadly adopted it needs to be better and it needs to be cheap. Publish and be damned. If the research is great the plumbing will proliferate at very little cost. If it does proliferate, you continue to research and innovate and profit above the new infrastructure, it is all good. If it does not proliferate..., well open source was not the problem!
Enterprise Ireland: open source can be a viable business model for shared infrastructure research. It is a world of constant iterative improvement. The profits are smaller but the rewards are greater because simply put, value shared is value multiplied.
In essence, open innovation puts the focus on execution rather than protection, if puts everyone on the front foot.
For a web architecture of the future look at Unhosted
At the root of the problem of siloed data and fragmentation (the database hugging phenomena) is the issue of ownership. Institutions have data that they don't really own because that data is of a personal nature. The collection of data is theirs, but not the individual components.
With Unhosted, the ownership problem is turned on its head. Users and aggregators of data only have a 'handle' (a URI) to personal data. A handle that is only useable with permission. Collections of data containing 'handles' can safely be shared. Granted, lots of issues need to be ironed out, but I think the architecture is on the right track and the concept is bang on.
Open source your research
Lots of what you do is plumbing. For new plumbing to be broadly adopted it needs to be better and it needs to be cheap. Publish and be damned. If the research is great the plumbing will proliferate at very little cost. If it does proliferate, you continue to research and innovate and profit above the new infrastructure, it is all good. If it does not proliferate..., well open source was not the problem!
Enterprise Ireland: open source can be a viable business model for shared infrastructure research. It is a world of constant iterative improvement. The profits are smaller but the rewards are greater because simply put, value shared is value multiplied.
In essence, open innovation puts the focus on execution rather than protection, if puts everyone on the front foot.
Oct 29, 2010
Independent FuseSource, a future of shared value
The future is bright for FuseSource and open source adoption, the challenge is to spread the word on shared value so more organisations can benefit.
At FuseSource, we are independent, we have a proven subscription based business plan and we have a clear message: "The experts in open source integration and messaging". A message that that is backed up by our Apache committers and consultants, many of whom are project founders. We are on the right track.
I think the growth of FuseSource is testament to the fact that enterprises are understanding a key benefit of liberal licensed open source:
Put simply, each deployment of Apache ServiceMix, Apache ActiveMQ, ApacheCXF and Apache Camel, contributes positively to the shared pool of knowledge about these products. At FuseSource, all enhancement and fixes are delivered first at Apache, so everyone can benefit immediately. A great innovation this week becomes the start point for a new deployment next week. There are no barriers to entry. We all get smarter together.
The reality is that open source consultants rarely repeat themselves, work done for one client is work done for everyone. It is a model of shared incremental improvement. It is constantly challenging work, but most rewarding and always interesting.
My hope is that more organisations, where information technology (IT) is not the core of their competitive advantage, will see the benefit of an open collaborative approach to infrastructure investment. The approach is simple: Use the same open source products as others, invest in those products, contribute back and reap the benefits of the contributions of others. Though we consider our selves individuals, when it comes to what we need computers to do, we are mostly the same.
If you work in health care, government or retail and have an IT problem, somewhere in the world some one is struggling with the same problem as you. You need not be alone, you just need to share a common language and join the community. Open source infrastructure can be that language.
Note: those organisations that use IT for competitive advantage are already on the open source band wagon, layering higher value services over existing open implementations, standing on the shoulders of giants. They just don't always have the same incentive to share.
At FuseSource, we are independent, we have a proven subscription based business plan and we have a clear message: "The experts in open source integration and messaging". A message that that is backed up by our Apache committers and consultants, many of whom are project founders. We are on the right track.
I think the growth of FuseSource is testament to the fact that enterprises are understanding a key benefit of liberal licensed open source:
Value shared is value multiplied
Put simply, each deployment of Apache ServiceMix, Apache ActiveMQ, ApacheCXF and Apache Camel, contributes positively to the shared pool of knowledge about these products. At FuseSource, all enhancement and fixes are delivered first at Apache, so everyone can benefit immediately. A great innovation this week becomes the start point for a new deployment next week. There are no barriers to entry. We all get smarter together.
The reality is that open source consultants rarely repeat themselves, work done for one client is work done for everyone. It is a model of shared incremental improvement. It is constantly challenging work, but most rewarding and always interesting.
My hope is that more organisations, where information technology (IT) is not the core of their competitive advantage, will see the benefit of an open collaborative approach to infrastructure investment. The approach is simple: Use the same open source products as others, invest in those products, contribute back and reap the benefits of the contributions of others. Though we consider our selves individuals, when it comes to what we need computers to do, we are mostly the same.
If you work in health care, government or retail and have an IT problem, somewhere in the world some one is struggling with the same problem as you. You need not be alone, you just need to share a common language and join the community. Open source infrastructure can be that language.
Note: those organisations that use IT for competitive advantage are already on the open source band wagon, layering higher value services over existing open implementations, standing on the shoulders of giants. They just don't always have the same incentive to share.
Aug 16, 2010
Reminder: JMS is client server infrastructure; update broker ∴ update client
Often we get the following question:
For the longer answer there are at least two things to consider:
Does an update to the openwire protocol version affect me?
The openwire protocol is the set of commands that is used to communicate between an ActiveMQ client and an ActiveMQ broker (and from broker to broker in a cluster scenario). The openwire protocol supports version negotiation such that an old client can negotiate the lowest common version with it's peer and use that version. As a result, in most cases, old clients can work as expected with a newer broker.
There are two potential pitfalls that you should be aware of:
Fixes or features that depend on the openwire version update
These are typically fixes that require additional information to be passed from the clients to the broker or vice versa. Some examples include the addition of a last delivered sequence id parameter to a consumer close command such that the redelivery count could be more accurately calculated. Another is the addition of a reconnecting flag to a connection command that allows duplicate suppression to be implemented consistently at the transport connection level. In some cases it is not obvious if an issue requires a protocol update without some consideration of the implementation, if in doubt ask on the activemq mailing list.
The ever increasing and incomplete version testing matrix
With every protocol version change, there are new additions to the client/server testing matrix. In ActiveMQ, virtually all tests assume a uniform openwire version, with the exception of a few that validate negotiation. The net result is that validation of the compatibility matrix is largely completed by the community. This works in practice but it is important to be aware of. If you are in doubt as to whether a particular scenarios will work across a broker version mismatch, be sure; ask the computer yourself with a little test.
In summary, If you update the broker, you also need to update the clients; or at least consider it!
If I upgrade the broker to version 5.x, do I also need to upgrade all my clients?The short answer is:
maybe, but error on the safe size and upgrade your clients if it does not cause too much disruption.
For the longer answer there are at least two things to consider:
- Does the reason for upgrade include the need for fixes that affect client side code? If yes, then obviously update all clients. (Issues of this kind typically focus on some aspect of the JMS API or consumer delivery semantics.)
- Is there an increment to the openwire protocol version? if so, does it affect me? read on...
Does an update to the openwire protocol version affect me?
The openwire protocol is the set of commands that is used to communicate between an ActiveMQ client and an ActiveMQ broker (and from broker to broker in a cluster scenario). The openwire protocol supports version negotiation such that an old client can negotiate the lowest common version with it's peer and use that version. As a result, in most cases, old clients can work as expected with a newer broker.
There are two potential pitfalls that you should be aware of:
- fixes/features that depend on the openwire version update.
- the ever increasing and incomplete version testing matrix.
Fixes or features that depend on the openwire version update
These are typically fixes that require additional information to be passed from the clients to the broker or vice versa. Some examples include the addition of a last delivered sequence id parameter to a consumer close command such that the redelivery count could be more accurately calculated. Another is the addition of a reconnecting flag to a connection command that allows duplicate suppression to be implemented consistently at the transport connection level. In some cases it is not obvious if an issue requires a protocol update without some consideration of the implementation, if in doubt ask on the activemq mailing list.
The ever increasing and incomplete version testing matrix
With every protocol version change, there are new additions to the client/server testing matrix. In ActiveMQ, virtually all tests assume a uniform openwire version, with the exception of a few that validate negotiation. The net result is that validation of the compatibility matrix is largely completed by the community. This works in practice but it is important to be aware of. If you are in doubt as to whether a particular scenarios will work across a broker version mismatch, be sure; ask the computer yourself with a little test.
In summary, If you update the broker, you also need to update the clients; or at least consider it!
Jan 18, 2010
ActiveMQ (prefetch and asyncDispatch) negative destination inflight values in Jconsole explained
While tracking down an issue for a customer over the past few days I noticed the inflight count for my destination in jconsole has a negative value. On closer inspection, I found that the value was fluctuating wildly with negative values before settling down again to a more reasonable positive range. I took a detour to investigate and it turns out this behavior is expected. The negative values are the result of prefetch and asyncDispatch, let me explain with a little note to self:
The use case included a pre filled queue with ~30k messages and multiple(10) consumers which dequeued a small amount(again 10) of messages before disconnecting and immediately reconnecting. In this case, a prefetch value of 10 is ideal, but with the default prefetch value of 1000, the broker is busy dispatching messages to the consumer long after it has decided to quit. In addition, with asyncDispath, while dispatch to a consumer is instigated by the broker, the actual delivery is delegated, to the broker transport connection worker thread. This means that the delivery attempts back up on the individual transport connections rather than slowing down the broker.
The destination inflight count is a measure of the number of messages that have been dispatched by the broker but not yet acknowledged by any consumer. On each dispatch completion by the worker thread, the inflight value is incremented. The decrementing normally happens on a message acknowledge. In the event of a consumer closure with unconsumed messages, the remaining value is decremented when the consumer closes.
This is the crux. From the broker perspective, on consumer closure, it has dispatched 1000 messages and got an ack for 10 so it needs to decrement the inflight by1090 990 (thanks for the correction Arjan). But from the perspective of the worker thread, busy doing the actual dispatch, it still has a lot of incrementing to do. The negative values arise from the consumer closure occurring before async dispatch is complete. When there are many concurrent consumers, the negative swing can be quite noticeable and quite large.
The good news is that this is perfectly fine, the books are kept in balance and there is eventual consistency. In addition, using either of a prefetch value of 10 or asyncDispatch=false ensures that the negative values do not occur as the broker is kept directly in step with message delivery to the consumer. In general though, the an appropriate value of prefetch is the correct solution if it is known in advance that a consumer will do work in batches.
The use case included a pre filled queue with ~30k messages and multiple(10) consumers which dequeued a small amount(again 10) of messages before disconnecting and immediately reconnecting. In this case, a prefetch value of 10 is ideal, but with the default prefetch value of 1000, the broker is busy dispatching messages to the consumer long after it has decided to quit. In addition, with asyncDispath, while dispatch to a consumer is instigated by the broker, the actual delivery is delegated, to the broker transport connection worker thread. This means that the delivery attempts back up on the individual transport connections rather than slowing down the broker.
The destination inflight count is a measure of the number of messages that have been dispatched by the broker but not yet acknowledged by any consumer. On each dispatch completion by the worker thread, the inflight value is incremented. The decrementing normally happens on a message acknowledge. In the event of a consumer closure with unconsumed messages, the remaining value is decremented when the consumer closes.
This is the crux. From the broker perspective, on consumer closure, it has dispatched 1000 messages and got an ack for 10 so it needs to decrement the inflight by
The good news is that this is perfectly fine, the books are kept in balance and there is eventual consistency. In addition, using either of a prefetch value of 10 or asyncDispatch=false ensures that the negative values do not occur as the broker is kept directly in step with message delivery to the consumer. In general though, the an appropriate value of prefetch is the correct solution if it is known in advance that a consumer will do work in batches.
Oct 15, 2009
Interpreting the ActiveMQ 5.3.0 SpecJMS2007® Result
Hot on the heels of the latest Apache ActiveMQ release, official SpecJMS2007® Results appear. ActiveMQ does 156 vertical and 60 horizontal. But what does that mean?
Some Background
SpecJMS2007® is a representative, long-running, comparative test. Let me take each of these in turn.
By representative, I mean that it contains a mix of business interactions that utilises point to point (or queue semantics) and publish/subscribe (topic) semantics. The message sizes vary within limits using random generators and there is a mix of non-persistent and persistent messages. All persistent messages are delivered and consumed within transactions. In reality, the interactions are based around a supermarket supply chain application which provides a rich tapestry for realistic actor interplay. Supermarkets querying suppliers, suppliers interacting with distribution centers and throughout, management in headquarters, keeping track of all dealings.
By long running, I mean that the scenario lasts a minimum of 30 minutes, excluding a warm-up period. The verification phase is based on periodic throughput and response time sampling during that period. In this way, the test verifies the sustainable load characteristics of a JMS Broker.
By comparative, I mean that the test artifacts and environment are completely specified such that results are totally reproducible. For example, if the broker implementation is swapped out from the ActiveMQ submission bundle, a comparable result for the same platform can be obtained. The platform (or OS and hardware configuration) must be maintained to produce comparable results. This focuses the comparison on the implementation of the broker, which is the intention.
Explain the Numbers
The numbers are seed values. They provide the base value or multiplier on which subsequent decisions like the number of destinations, quantity of messages etc. are determined for a given test run. An increase in the seed value has a cascade effect on the overall load that is placed on the system. If the seed value is too high, the overall load will result in a failed test. Either because of unacceptable throughput variance or because of response times exceeding predetermined ranges. To pass the test for a given seed value, all response time and throughput expectations must be met.
The vertical and horizontal qualifiers refer to the SpecJMS2007® workload topologies. The topologies are not directly comparable because the seed multipliers have different effects in both topologies.
Vertical
As the seed value increases, the vertical scenario aims to increase the number of messages that are processed for a given number of destinations. So in the supermarket supply chain parlance, this means increasing the quantity and variety of stock that is maintained and the frequency of replenishment of said stock. The number of supermarkets and suppliers etc. remains constant as the base seed value increases. In this way, the ability of the broker to deal with increased load on existing destinations is explored. Another way of looking at this is that the depth of the destinations rather than the number of destinations is increased.
Horizontal
As the seed value increases, the horizontal scenario aims to increase the number of destinations while using a fixed load of messages. This corresponds to adding more supermarkets, suppliers and distribution centers. The quantities of stock and the frequency of replenishment is constant. In this way, the ability of the broker to deal concurrently with large numbers of destinations is explored.
In short, the numbers are meaningless in isolation as they are the units of SpecJMS2007® performance measurement and these units have no real-world corollary. Where they are useful is when used in comparison with another run of the SpecJMS2007® test using a different JMS implementation or with some broker configuration tweak.
For example, there are two platform variants of the results, one with Hyper Threading(HT) enabled and the other with HT disabled. The effect is significant indeed, with the vertical successful seed value going from 138 to 156 and horizontal from 52 to 60. So turn HT on!
Some Background
SpecJMS2007® is a representative, long-running, comparative test. Let me take each of these in turn.
By representative, I mean that it contains a mix of business interactions that utilises point to point (or queue semantics) and publish/subscribe (topic) semantics. The message sizes vary within limits using random generators and there is a mix of non-persistent and persistent messages. All persistent messages are delivered and consumed within transactions. In reality, the interactions are based around a supermarket supply chain application which provides a rich tapestry for realistic actor interplay. Supermarkets querying suppliers, suppliers interacting with distribution centers and throughout, management in headquarters, keeping track of all dealings.
By long running, I mean that the scenario lasts a minimum of 30 minutes, excluding a warm-up period. The verification phase is based on periodic throughput and response time sampling during that period. In this way, the test verifies the sustainable load characteristics of a JMS Broker.
By comparative, I mean that the test artifacts and environment are completely specified such that results are totally reproducible. For example, if the broker implementation is swapped out from the ActiveMQ submission bundle, a comparable result for the same platform can be obtained. The platform (or OS and hardware configuration) must be maintained to produce comparable results. This focuses the comparison on the implementation of the broker, which is the intention.
Explain the Numbers
The numbers are seed values. They provide the base value or multiplier on which subsequent decisions like the number of destinations, quantity of messages etc. are determined for a given test run. An increase in the seed value has a cascade effect on the overall load that is placed on the system. If the seed value is too high, the overall load will result in a failed test. Either because of unacceptable throughput variance or because of response times exceeding predetermined ranges. To pass the test for a given seed value, all response time and throughput expectations must be met.
The vertical and horizontal qualifiers refer to the SpecJMS2007® workload topologies. The topologies are not directly comparable because the seed multipliers have different effects in both topologies.
Vertical
As the seed value increases, the vertical scenario aims to increase the number of messages that are processed for a given number of destinations. So in the supermarket supply chain parlance, this means increasing the quantity and variety of stock that is maintained and the frequency of replenishment of said stock. The number of supermarkets and suppliers etc. remains constant as the base seed value increases. In this way, the ability of the broker to deal with increased load on existing destinations is explored. Another way of looking at this is that the depth of the destinations rather than the number of destinations is increased.
Horizontal
As the seed value increases, the horizontal scenario aims to increase the number of destinations while using a fixed load of messages. This corresponds to adding more supermarkets, suppliers and distribution centers. The quantities of stock and the frequency of replenishment is constant. In this way, the ability of the broker to deal concurrently with large numbers of destinations is explored.
In short, the numbers are meaningless in isolation as they are the units of SpecJMS2007® performance measurement and these units have no real-world corollary. Where they are useful is when used in comparison with another run of the SpecJMS2007® test using a different JMS implementation or with some broker configuration tweak.
For example, there are two platform variants of the results, one with Hyper Threading(HT) enabled and the other with HT disabled. The effect is significant indeed, with the vertical successful seed value going from 138 to 156 and horizontal from 52 to 60. So turn HT on!
Jul 14, 2009
Apache ActiveMQ Out Of Memory!
Apache ActiveMQ is adaptable and configurable. A large part of its popularity is due to its flexibility. However, it comes with a default activemq.xml configuration file that cannot possibly suit everybody's needs. The default configuration is a compromise between memory utilisation, low latency and high throughput, with a smattering of feature demonstrations. In all, it is probably too much for one configuration file, but that is another issue that is in part addressed in version 5.3.0.
With the current defaults, it is relatively easy to push the broker's heap memory utilization past the
What to do?
Well Google is your friend but there is also the ActiveMQ FAQ and particularly the entry that deals with the likely causes and relevant configuration that can alleviate ActiveMQ OutOfMemoryError Exceptions.
In short, the answer is nearly always configuration and the intent is that the OutOfMemory FAQ entry will provide a comprehensive reference for the relevant options. Let it be your first port of call.
With the current defaults, it is relatively easy to push the broker's heap memory utilization past the
-Xmx512m heap limit passed to the JVM in the start script. When that happens the broker begins to fail in various places with java.lang.OutOfMemoryError: unable to create new native thread or java.lang.OutOfMemoryError: Java heap space.What to do?
Well Google is your friend but there is also the ActiveMQ FAQ and particularly the entry that deals with the likely causes and relevant configuration that can alleviate ActiveMQ OutOfMemoryError Exceptions.
In short, the answer is nearly always configuration and the intent is that the OutOfMemory FAQ entry will provide a comprehensive reference for the relevant options. Let it be your first port of call.
Subscribe to:
Posts (Atom)
