BPM Futures

Let data make that process dance

I was recently discussing BPM agility with a senior IT manager, when the topic of configuration files came up. “Many of my colleagues don’t like them” he said “we have hundreds of them, and they don’t like changing them because they don’t know what the consequences will be”.

It’s easy to smile at this, but practically all guardians of customised systems are in a similar position. The reason that I’m writing about it is because config files (or config data, to generalise) can be a powerful way to drive change to BPM systems if used correctly, because data can be significantly easier to change and deploy than code.

What sort of data might one want to externalise in order to improve BPM agility? A few examples:

  • Field labels
  • Text to be included in reports/alerts
  • Route/path ids and sub-process ids (to permit config-driven re-routing)
  • Data relating to rules (‘if value of loan > $1000 then’ will obviously be more flexible if the value – $1000 – is a variable) if the rules are not already externalised through a rules engine (which would be better still).
  • Data that drives which cases/process instances should adopt changed config values. Should changed values be picked up by closed or archived cases? Should they only apply to new cases, or cases with particular characteristics?

As the last point suggests, there is an important system design aspect to this, regarding when config data should be (re-)read and how much granularity is required to control change impact.

And of course, the data to be held will vary widely, depending both on the business processes and the BPMS used.

For this data driven approach to deliver significant improvements to BPM agility the mechanisms used must be thoroughly tested – part of ‘Agility Testing’. Unless one can be confident that changes to the config data will result in predictable outcomes, abstracting it will provide few benefits, since changes will once more need to be system tested, delaying deployment.

One significant source of comfort for the hard-pressed project manager is that Agility Testing can be carried out after the initial go-live. Not perfect, obviously, but a promise that ‘the system will become easier to change after the second deployment’ will put you ahead of many BPM solutions.

So are config files, as such, the best way of delivering the config data into production? Well, no. Config files are hard to add CRUD functions to (and so are often edited in Notepad – easy to make a mistake) and must be deployed (albeit it’s a very simple deployment).

Using a relational database (RDBMS) helps a lot, in terms of basic validation, and if already in use on your project it will be easy to piggyback on existing or planned resilience and recovery features. However, without a user interface (UI), the standard method of changing an RDBMS is via a stored procedure (for audit and repeatability reasons) and this, once more, will require testing and deployment.

A custom UI would be best. It needn’t be pretty – it will only be used by Admin users or Prod Support so some shortcuts may be possible in terms of corporate standards. However, it will need to be able to provide Read/Update (possibly Create/Delete) functions for quite an assortment of data, so some extendable way to segment the data, such as tabbing, will help. Most importantly it must include an audit log and, ideally, a function that will import an existing audit log, making the changes included.

The reason for the audit log import is procedural. Even though no technical issues should arise from changes made via the UI, there is still potential for the Business to have erred in their change request. So an initial deployment into a User Acceptance Test (UAT) environment may well be required. The UI may be used to directly change the UAT config data – indeed this should be so easy that it could be done with a business representative alongside. Once the Business is satisfied with the result, the audit log can be deployed into Production and run against the Production UI (as an import) with no further testing required, since the resulting Production changes will necessarily be identical to those in UAT.

The BPM project steering committee will need to assess the value of this sort of approach, since it clearly involves a cost. The degree to which config data can be used to drive agility in a BPM solution will vary, as will the value that the Business places on it.

And of course all of this works best within an A-BPM (Agile BPM – here today, Gartner tomorrow!) development framework.

Happy trails…

“Agile” – time for a change

Back in the early 90s we used to demonstrate workflow by building a 3-step ‘Leave Application’ process from scratch, run it from the user perspective, change it, then run it again. The message, reinforced by our patter and marketing collateral, was clear – workflow was Agile. And you can read the same from every BPM vendor today – BPM will result in (or facilitate, support – pick your own weasel word) Agility.

The reality is rather different. Most operations managers have to prioritise the changes they need to their BPM system and these typically get delivered according to a release schedule that is measured in weeks rather than hours.

Why? Well, there’s the complexity – of real business processes that are usually much more complex than their users first thought, defined in tools that are smart but not quite perfect (think workarounds), including multiple integration points (so we may need to change the Java code too), and a user interface that is shared with other systems and environments. So the typical end result of a single process implementation is something that is quite hard to ‘get your head around’ and requires special skills to change. For a multi-process deployment, this gets even harder.  And this complexity of interacting components results in genuine risk – of developer error, of user error (in terms of clearly thinking through what is required), and of deployment error.

These risks are typically mitigated through a series of tests. Systems Testing is specifically designed to test that the changed components work together as expected by the developer. Regression testing will test that the rest of the system has not been impacted by the change. Acceptance testing ensures that the user gets what they (thought they) asked for, and gives them a chance to change their minds. And post-implementation verification testing validates that the smorgasbord of changed components that has been deployed hasn’t destabilized the live system. All of this testing and deployment activity takes time and is more efficiently carried out on batches of process changes, rather than one change at a time. The actual deployment may well need to happen outside of working hours, too, as a further risk mitigation strategy. It therefore tends to happen every few weeks, not daily.

Is this inevitable? I don’t think so. After all, there are some changes that are always carried out swiftly – on the same day or, at worst, overnight. Adding a new user, complete with a required permissions profile, will have significantly complex effects – not only on that user’s system access, but also on reporting, enquiries, and supervisor access – but is typically carried out within hours. Why shouldn’t the same apply to process changes?

The answer, today, is that your current BPM solution – typically a BPM product that has been configured, customized and extended to meet your requirements –will not have been designed to support true Agility.

Does any of this matter? Well, think back to the much-derided paper-based process that the BPM system replaced. The process was largely determined by the contents of a tick sheet stapled to the front of the manila folder that contained the case documents. The users knew the process rules, which were based upon what was ticked and what wasn’t.  So changing the process involved changing the tick sheet (thanks, MS Word) and giving new instructions to the team. It could be done in hours. Sure, the end result was much more error prone, less efficient and lacking in MIS. But are we currently trading enormous improvements in all of these, when we deploy BPM, for a loss in agility?

I believe this to be the single biggest challenge to BPM, with truly agile BPM providing potentially one of the most radical changes that technology can contribute to business processing. As an industry, can we rise to the challenge? What did The Man say….? Yes We Can?

To be continued. Happy trails….

BPM – the future starts here

Having read a few other ‘first blogs’ there seems to be a tradition of keeping the first one light. Well, I’m not going to do that – I want to really get stuck into a current preoccupation – what is the future of BPM? Now I admit that I’m biased to a positive view, having worked in BPM (nee workflow) since 1990, and the title of this blog ‘BPM Futures’ also indicates a certain confidence.

However, having been focused pretty much exclusively on business unit, account and project management for the last 5 years (mostly relating to BPM, btw), I feel in need of a domain refresh. So this is it – I hope you’ll also find it of some interest and use.

So why do organizations buy BPM, rather than other solutions available in the global software market? My answers are listed below, along with some observations that will provide recurring topics for future blogs.

1. The Spot Solution “We want to manage our process better, whilst keeping most or all of the standard data in systems other than the BPM system”.

Comment – essentially a tactical view, this generally happens because the client has calculated that it’s cheaper/easier to use a BPM system than to extend their existing system(s) for the purposes of process management.

Examples- streamlining a home loan approval process that straddles an enterprise customer information system  and  a ‘stovepipe’ home loans system;  managing an accounts payable process in which several thousand employees are occasional users and only half a dozen users – in the a/p section of the accounts department – need to use the a/p module of the accounting system itself.

Challenges and alternatives – if the cost of replacing the existing system were acceptable, most newer ‘core’ systems – such as home loans and a/p – include workflow (if not BPM) functionality appropriate to the specific application, eliminating the need for integration.

Opportunities and the future – merger and acquisition (M&A) activity, if nothing else, will maintain this market for BPM. And of course the gradual adoption of industry-standard interfaces and standards by core system vendors will make the integration challenges easier over time. That said, there is plenty of room for improvement in the BPM solutions offered.

2. The Strategist “We see benefit in using the same process management tool for a wide range of processes”.

Comment – Why? Because it makes skills management easier and allows standardization on common process management features such as process definition, version and release management;  data management/reporting; full cycle process improvement; process simulation. At a more strategic level, BPM is seen as protecting process assets from change at the transaction system level, in particular providing flexibility following merger or acquisition.

Examples – there are many large organizations that have invested heavily in BPM and deployed it very widely indeed, with enterprise process management as an explicit goal. Particular examples exist in retail banking, life insurance, wealth management, P&C insurance and telcos.

Challenges – naturally big, enterprise solutions raise the largest problems; agility/ease of change of processes; adherence to standards; the difficulty of full lifecycle process improvement in a highly diverse technical environment; the role for rules management; and managing the explosion of UI requirements that arise from enterprise BPM deployment.

Alternatives – the current wave of core system replacements in the banking industry and the increasing adoption of eTOM-based enterprise telco solutions seem to pose a medium-term challenge to BPM (and, perhaps to a lesser extent, SOA) as a separate industry, though the concepts should live on within the replacement products.

Opportunities and the future – not only core systems, but also enterprise BPM systems are potential subjects for upgrade, particularly as the limits of ‘first generation enterprise BPM systems’ become apparent to user organizations. There are a number of areas that can benefit from improvement and, at least until core system upgrades are complete (at least 10 years away), demand exists to pay for them.

3. The Document Manager “We started to look at document management – scanning our incoming mail and working from this and our incoming faxes through an image viewer, rather than paper – and realized that workflow and, better, BPM, was a significant value add”.

Comment – this is a common sense and frequently used reason for BPM adoption, particularly in a departmental and/or small/medium enterprise context. The current strategic challenge for document management product vendors – increasing commoditization and therefore lower margins – provides opportunities for solution/service providers and customers alike.

Opportunities and the future – whilst this customer perspective starts a little differently (with document management) it soon returns to the familiar territory of specific process management (what are the scanned documents being used for?) and interactions with other systems. So the challenges are similar to ‘The Spot Solution’ and others.

4. The Administrator “We’re looking for a cheap and cheerful way to improve our simpler, administrative processes that our business requires but none of our core systems provide”

Comment – classic examples used are leave processing; timesheets; expense claims; many aspects of employee on-boarding and maintenance, such as security card applications; and so on. Also processes needed to support short term projects often fall into this general category.

Of course even ‘simple’ processes like these would ideally include integration (do we need to see receipts before approving expense claims? shouldn’t timesheet and leave processes interact with the payroll system?). Most BPM systems will provide the technology to provide this integration, though the services required might make the overall business case tough.

Opportunities and the future – this has always been a popular area for workflow, particularly if integration can be ignored (ie worked around by a human operator). Today the relatively simple functionality required combines naturally with BPM delivered through the web as a service, with power provided by cloud computing. No doubt it will soon be as easy and cheap to build an online process as it is today to build a web site (just start with a template…), providing exciting opportunities for a huge number of users and a (very?) few cloud-based suppliers, whilst soon removing this as a market for traditional workflow/BPM solution providers.

5. The Case Manager “We need a Case Management system”

Comment – this is an interesting one. BPM products have frequently been used as part of, or even as the core product for, case management systems. The reason being the obvious one, that BPM products include much of the functionality required for case management, plus the promise of some interesting ‘extras’ (simulation, full cycle process improvement) and in addition many have a significantly larger user base than specifically ‘case management’ products.

Challenges – there are many points of divergence between case management requirements and those of other processes, and many of these can be seen as gaps in BPM product functionality. Two examples  (1) case management workflows tend to mix status (‘state’) and workflow process rules (so a case may be worked on by a case worker, who may carry out a range of actions, some of which include their own workflows, such as a referral to a colleague; once a defined status/state is reached the case must be sent for approval (another workflow); once approved, the case is returned to the case worker who now has a different range of permitted activities, because the state has changed. Standard ways of representing process flows, including BPMN (BPM Notation – think swim lanes), are very good for workflows and equally clumsy or long-winded for defining and illustrating state changes – ‘state transitions’, in the jargon); (2) case management puts a lot of emphasis on data relationships (so in a legal case a law firm may need to record many clients, witnesses, third parties, etc in respect of related cases – BPM systems tend to most easily support only simple data relationships, at both data storage and UI levels, requiring customization to support more). These ‘gaps’ provide opportunities for service providers to craft custom solutions today, and product vendors to extend their products tomorrow.

For the best summary of the special features of case management I have seen recently, check out the following, published here by the ever-readable Business Process Trends:


Yes, it’s produced by a vendor – Singularity – but it’s a great contribution to the literature on the subject, nonetheless.

Opportunities and the future – demand for, and therefore interest in, case management appears high at the moment. This probably reflects its concentration in the Public Sector, where every new government initiative requires administrators/administration and, usually, case management. And there is no equivalent of eTOM on the near horizon, as far as I’m aware (any ideas from anyone else on that?).

So the market for case management systems that manage case data, processes and reporting flexibly and reliably appears secure for now. Improvements are, again, needed. The recent signs of movement in the OMG, triggered by a draft RFP from the guys at Cordys, might result in some standards in this area, and/or might stimulate a product vendor to push ahead with some real innovation.  Lots of topics here for future blogs…

If you read this far, well done. Not an easy read, and I can assure you, not an easy write either!

Happy trails……..