In my previous post I mentioned that I’d be developing a non-trivial, real-world application from scratch and approach it in the same way I would if I were doing it for a paying client. The first step in that direction is to gather requirements and define the project:
- What is the main purpose of the project and what need does it satisfy?
- Are there real and artificial limitations placed on the project?
- What technologies will be used to satisfy the requirements? Why?
There are obviously loads more questions that we need to ask, and answer. But lets start with these three. The rest of this post will provide a broad outline for the project, where I’m going to try to set the boundaries, and why I chose this specific idea to tackle.
The Problem of Workflow
Since my early days working with the Oracle database, all the way into my tenure with Oracle APEX, workflow has been something of a conundrum. When most people see or hear workflow it usually goes one of two ways: They think of something specific to their business and therefore fairly concrete (in their minds at least) or they think of something wildly generic that will suit every case that ever existed and ever will exist.
As a long time consultant programmer, every time I discuss workflow with a client it usually goes something like this.
Client: We need some sort of workflow engine so that we can track the progress of Process A through its multiple steps.
Me: Ok, I’d love to understand more about the steps in this process, how complex is it?
Client: Oh, very complex. There are multiple steps, various people involved, various decision points that may change the outcome and we have restrictions on how long the process can take.
Me: How “static” is the process? Do the steps and process change over time or has the process been fairly stable?
Client: It’s fairly stable, but we’d like to be able to react quickly to new business drivers that may push changes to the process. We pride ourselves on being a nimble company and that’s what keeps us competitive.
Me: Understood. Sounds like you need a full-fledged workflow engine that allows you full flexibility to adjust process definitions. There are a number of commercial solutions out there. Are you interested in looking at any of them?
Client: We’ve actually done a bit of research and the commercial solutions are either too complex, too expensive or too hard to integrate. We’d rather have something written that suits our specific needs.
Me: [ … silence with a blank expression … ]
At this point it turns into the age-old discussion of the value of buy vs. build. The cost of maintaining home-spun software and the value of a supported product. If they truly choose to go down the “roll-your-own” path, then there is inevitably the discussion of re-inventing the wheel.
In my experience, more times than not, the client abandons the entire idea of a flexible workflow engine and ends up hardcoding the process steps into whatever platform they’re using to develop. I can’t tell you how many clients I’ve built code for that, as soon as the process steps need to change, they’d have to rip apart and rewrite.
There have been a few clients over my 30+ years of doing this that, while they didn’t want to buy a huge workflow package (feeling it was overkill), did want truly flexible workflow. For these customers we’d develop something that was flexible enough to suit their business model, but it was so tightly integrated to their core system that it really wasn’t usable by anyone else in the company.
One (1!) customer let me convince them to implement PL/FLOW and integrate that into their system. This worked well, and they were happy with it for quite a while. Then when they wanted to integrate it with another system, they were so confounded by the whole thing, and because they weren’t able to get support (and didn’t want to pay me to help them) they ripped it out. I’m still not sure what they did to replace it, but alas it is not my problem now.
What do we take away from all this? Workflow is not straight forward. It’s hard – hard to understand, hard to implement, hard to integrate into existing systems. It’s just hard. So this, in essence, defines the answer to the first question I outlined above. What is the main purpose of the project and what need does it satisfy:
Build a flexible, understandable and well documented workflow engine that is easy to integrate into any application and that will serve the majority of people’s needs.
Provide an easy to use GUI that allows users to create and change workflows visually.
Provide a REST based API layer that allows external applications to start, progress, stop and query the state of a given workflow.
Provide several reference implementations in different technologies to show how the workflow API layer can be leveraged.
Make it freely available to anyone who wants it.
Don’t try to boil the ocean!
Lofty goals indeed, but that’s the gist of what I’ll be trying to do here.
Building Real Workflow or Re-Inventing the Wheel?
As I mentioned above, there are quite a few commercial, scalable and flexible workflow products out there on the market. Just Google “Workflow Engine” and you’ll get north of 6,200,000 entries. Google “Free Workflow Engine” and that number gets cut in half. Still a significant number, don’t get me wrong!
So why build Yet Another Workflow Engine1? There are a few reasons.
- To Simplify – As I said, workflow is complex and is often made even more complicated by systems that try to implement a solution that can be all things to all people. My experience is that the greatest share of workflow requirements, while not exactly simple, aren’t rocket science either.
- To Educate – Not only those who are reading this, but myself. Workflow is something that most people understand the concept of. But how to implement it, how to use it, what it technically entails to develop – maybe not so much. This is, after all, the whole reason for this blog series!
- To Push Myself – I can happily say that I know a lot about Oracle in general. I’ve been doing it since 1988 and have picked up a fair amount of knowledge on the way. However there are many dark corners of the Oracle toolsets that I have never visited. Part of the reason I’m doing this is completely selfish. I want to learn more about my chosen tools of the trade.
This brings us to the second question in the list above. Are there real and artificial limitations placed on the project?
First lets address the real limitations:
- While I may, nay will, take inspiration from external sources, everything I will be doing will be using, as much as possible, pure Oracle technology and will be coded from scratch. This will limit some design and implementation decisions that, in a green-field project devoid of technological baggage, may point toward different ways of doing things. That’s fine. We all know why we’re here. Lets Move on.
- With a goal of creating something that will run on just about any Oracle platform, I shall not use any Enterprise level features. That means I’ll be developing in the shallow end of the database pool, using Oracle SE or even Oracle XE.
- This is not an academic exercise to try to see how fancy or clever I can be. In the end, the bloody thing not only needs to work, but needs to be useful. I have long-held the mantra “Implement the simplest thing possible to get the job done” as a design mandate. That, along with the limitation of using only Oracle technologies, will obviously shape the project from beginning to end.
Now what about those artificial limitations:
- Probably the biggest of these, and the only one worth mentioning, is my time. Not only do I have a day job doing cool things at Oracle, but I also have a life. I won’t be locking myself in a developers dungeon until the code is done. I’ll be doing this as I get time and I’ll likely be learning along the way. That means there will be gaps in productivity while I bone up on something like how to build a proper JET application or how VBCS works. Bare with me – I’ll be sharing my insights (and pain) along the way.
1 Hmm.. YAWE might be a great name for the end product! I shall build a workflow system and it shall be called YAWE!
It’s an Oracle thing…
The last of the three questions is probably the easiest to answer: What technologies will be used to satisfy the requirements? Why?
Why? You have to ask?
Now having said that I can take a swipe at what I’ll likely use for each major area:
- Database Design – Oracle Data Modeler
- Screen Design – Balsamiq Mockups (one of the few non-Oracle tools I’ll probably use)
- Code Editor – Oracle SQL Developer / SQLcl
- Code Repository – GitHub (https://github.com/DougAGault/AllTheThings)
- API Documentation – Swagger.io
- Core Engine – Oracle Database and PL/SQL
- Workflow Builder Interface – Oracle APEX and ORACLE JET
- APIs – PL/SQL and Oracle REST Data Services
- Client Applications – APEX, JET, VBCS, ???
- Communication standard between Client and Core – JSON
These are obviously open to change and influence by you, the reader. But I think that’s a reasonable estimation of where we’re headed.
Workflow is my chosen project. Its been an idea that’s been stuck in my head like a bad 80’s song for years. Now I finally get to set it free.
While you’re not likely to dissuade me from going down this path, I’d love to hear your thoughts on this choice and the topic in general.
Until next time…