You can find it for yourselves on the LI Groups (mostly the Automation and Process Control groups), although that means that you may have to join them,
Why do so many Automation Projects go wrong?
From the Automation Group
This is a very broad subject to which there can be many different responses.
From my personal experience, I have seen automation projects fail for the following reasons :
1. Incomplete brief from the client.
2. Poor selection of equipment. Lack of experienced personnel making unsuitable
choices. Most common mistake is using complicated solutions. KISS principal is
paramount here.
3. Poor planning and scope of work. Again, lack of insight into what is required often
leads to an un-satisfactory conclusion
A general comment here : my credo has always been that I have always asked someone more knowledgeable than myself when in any doubt. Many automation practitioners believe they would disgrace themselves by doing this. The opposite is however true and normally leads to an improved solution.
----------------------------------
Following are my observations:
1.Lack of proper understanding of the "process to be automated" by the automation engineer
2.Improper sizing of Final Control Elements by the owner or their engineers.
3.Additions or revisions of I/O or functional requirements by client during FAT, pre commissioning or post commissioning stages or at SAT
4.Cutting corners by automation company to secure orders!
5.Foolish confidence on the part of automation engineer that he can conquer anything by his programming skills!
7.Wrong assumption by the client that "PLC or DCS" can do anything and everything and "Functional scope" can be expanded at any phase of the project!
----------------------------------
Although the above comments describe some key issues for failures, on a high level there is a serious gap with Automation Managers in adhering to basic project management principles. Most of the projects are executed merely depending on the historical information of the previous projects and the experiance of the project manager or lead rather than analysing facts. I would say that emphasizing on project management guidelines would highly reduce risk and calling out for project failures
----------------------------------
The points raised are excellent and show in-depth understanding of how easily things can go wrong : Taking the first of these, it IS so easy for an outsider to fail to come to grips with the dynamics of the Process when software and indeed, hard-ware are designed and chosen. One should always get an experienced Process professional skilled in the actual Chemistry of the plant to be automated to check out one's ideas. many of these could be flawed.
Then, despite the plethora of Valve sizing software available, my own preference has ALWAYS been to let the Valve manufacturer specify the actual size and trim of the valve in question. ( Fill in the valve sizing data form he provides ) Then if it does not work as per designe criteria, there is always a fallback position. I have worked in so many plants where my pre-decessors have installed oversized valves, that I am very wary of falling into this trap.
Once the design is accepted, the client needs to accept that "Design Freeze" has been reached ( formal milestone in the Project schedule) Any required divergence from this will cost HIM money.
The other points are equally relevant, but I will let others share their views,
regards,
----------------------------------
• It's the lack of clear objectives and definition of the technical and commercial requirements. All the process elements, ranges of tolerances and the variables must be identified and addressed along with the costs and the timing for implementation to ALLOW for success - execution is another matter entirely!
An important point not touched on yet is the long term commitment the customer has to make toward automation. Even the best engineer cannot predict the changing business rules or develop a perfect design. This is especially important moving from a manual system for the first time as the entire business model and resourcing requires changing too.
I've seen many 'system failures' blamed on a few small component, miss-scoped elements or training issues that remain undefined and discredit the entire system.
I agree with Charles around the KISS principle and don't over engineer too soon you are sure to miss-scope and also the Design Freeze however... I recommend starting your improvement program from day one to allow all those scope changes to be collected and distilled. I'm a strong advocate of Agile PM techniques to allow the system to grow into the business rather than spend months fighting over spec's that will never stand up in court anyway...
----------------------------------
From my experience, transforming the Project requirement into a System design is the key aspect of success for any Automation project.
An Automation system design should balance 1.‘Hardware modules (Instruments, motors/actuators, sensor/signal interfaces, etc…)’, 2.‘Software modules’ (UI, signal/image processing, control logic, etc…) & 3.‘Mechanical modules ‘(robot arm, fixtures, etc) in an accurate proportion.
A simple example; while designing a Vision system, when a limitation is hit on achieving a positional accuracy for a mechanical arm in positioning the Part under Camera’s field of view, the designer should know to what extend this deviation in position can be taken care by a suitable Camera selection or by Image Processing, with which an efficient system can be designed to meet the process/project requirement. In many instance, the Automation project will lead to failure only because of bad design or lack of expertise for designer.
The designer should have good experience/knowledge on all of 3 core components (mentioned above) to design an efficient automation system.
Sometimes the respective (Software, Electrical & Mechanical) engineers would step into the project development with assumption/over confidence and without having clear understanding on complete system requirement or other system modules that would lead to project failure in later stage of development or sometimes on deployment at the customer site.
From the Process Control Group
you only know what you know when you know it.
----------------------------------
when we do not give much attention to the fine details ,projects becomes difficult
----------------------------------
From my experience, which is not breweries, there are a couple of main areas that have caused automation projects to be less successful (I hate to say fail). Poor planning up-front with no real vision to what the final product should be (not only look like but how it should function), is one of the first things that comes to mind. The other is not getting all the right stakeholders involved from the beginning to get their buy-in AND give them a sense of ownership.
----------------------------------
I have learned that any project can be managed correctly and will lead to the desired and working result if a good functional design is made and discussed with the client (or made by the client and discussed with the supplier). Also from this document a complete test protocol can be derived quit simple, so testing and approving an application is also easy and complete. The problem is that both supplier and customer have to really read and understand the document.
The document itself can be written in any language and does not have to be technical. It is more important to describe the complete behaviour including all (un)known situations which can be presented to the automation environment.
The choice of hardware and software to realise the solution should be made when the functional design is complete. This can lead to a technical design (for instance to break the software into pieces that can be handled by the team or to design a standard structure to use in all equipment).
In this way both customer and supplier have to invest time in the start of the project but it will definitely pay off in the end of the project (less problems, less discussions, less bugs, etc.). The project will run in a realistic time schedule.
I am afraid the article is pitched as a sales exercise and does little to elaborate on any best practice methodologies, software engineering or requirements engineering that are the silver bullets used to tame the wild beast that every project has the potential to become. Knowledge of an industry alone doesn't give you the skills to actually execute good projects!
If you want good pointers then read the software engineering books by Ian Somerville and Roger Pressman. Look at SWEBOK (Software Engineering Body of Knowledge) an excellent resource!
Don't forget to look at new methods such as SCRUM. Embrace requirements engineering and use good design practices coupled with sound quality assurance and project management.
Following the above doesn't necessarily guarantee a positive result, there is always the totally unexpected but.... ignoring the above will almost certainly cost you time and money and maybe the whole project too!
----------------------------------
I agree in part, although to be fair to René Leven his point is that the Process Expertise that his company can provide (in a specific area, brewing) is beneficial, and I don't disagree with that.
Also the software engineering methodologies that you mention are, like most of them, based in the IT world, and not automation, and I strongly believe that the two, while sharing some things, are different.
I mentioned my old "What goes wrong with batch process control system projects? "
www.controldraw.co.uk/presentations/frspaper.htm
It is a short version of a longer paper, and yes it promotes my software ControlDraw - which you could call a CASE system for requirements engineering but unlike IT systems like UML is actually orientated around automation. It is not orientated around a particular solution.
----------------------------------
Industry knowledge may appear to have benefits but it also has limitations as to its use. Knowledge gained by working on a project for most clients is often proprietary and cannot be readily divulged or utilised on another client's projects without first gaining the consent of the owner of the information often regarded as the intellectual property of that company. We know that most industries are competitive and as a result they defend their process knowledge with a passion and rightly so. After all companies often spend millions of dollars to obtain and perfect that knowledge.
I looked at the question of IT versus IAC software engineering methods a few years ago whilst writing my dissertation "Requirements Analysis in Industrial Automation & Control". I spent a couple of years researching similar topic(s) to the one driving this discussion and the results of my work are contained within that document. If anyone would like a copy then please drop me a line?
Since then I have used The UML as a design tool for a mining project in Australia and Structured Analysis on a project for Petro-Canada with great success. I modeled all of the PLC requirements for the mining project in The UML first and then developed the code templates in GE-Fanuc RX3i PACs. I have an example of this approach if anyone would like to see what I did?
I like ControlDraw and I think it is the best tool yet for modelling IAC specific projects and would recommend to anyone reading this article to try it. But at the same time I would also advocate using some of the methods described within the books I mentioned. They are written to cover IT and IAC related methodologies.
----------------------------------
And I think you used ControlDraw 1 in 1999. Believe me it has been improved beyond what you might even have imagined since then, and yes I have looked into UML and Structured Analysis methodologies a lot in the process of developing version 3.
Re the 'intellectual property' that you find in various projects, it is true that companies often defend it. I have hundreds of models from such projects. But when it comes down to it an awful lot of it (95%+) is virtually the same as other people do.
I would very much like to see your dissertation, you know my direct mail address.
And if you are ever nearby drop me a line.
----------------------------------
My experience is based on spectroscopic/photometric process control.
First 10 years I had sometimes to learn the hard way and and I could see also at
other vendors of such instrumentations solutions had difficulties.
Sometimes it looks that there is a roll back in technology for some years, because the user
have so many bad experiences with few solutions.
The reason is, that big and small vendors try to offer a "product" or solution "as is" and this has to cover the clients application. This can not always works and depending the chosen product setup the risk for mismatching is higher or lower.
For this reason I left the original way to be a "instrumentation vendor" with certain products for sale a certain number every month or year .
Now I have a "building set" of the most critical things I have learned and I add around engineering, consulting, feasibility testing and assistance to choice the right components from third part vendors.
My experience now is, less trouble and mismatch!
Means more successful finished projects!
----------------------------------
My experience is in the APC area and competing Honeywell, AspenTech, Yokogawa, and others is the bidding process is all smoke and mirrors. Clients have no idea what is needed in the bidding for a successful outcome.
----------------------------------
"is all smoke and mirrors." sounds tough, but confirms my opinion
if "dinosaurs" try to rule inovation ...
----------------------------------
To develop a successful automation project should be first of all a functional description and a clear idea of the process to be automated.
After this have to develop the philosophy of control and select the technology that will make the solution. This technology must have a local representative (or at least on the same continent).
Along with a good electrical installation with the electrical protections well calculated should be made a good simulation of the system and a final test of acceptance with the participation of the client.
Prior to the commissioning of the equipment should be thoroughly reviewing installations and wiring terminations.
Other thing you don`t see in the books is that the more the client get involved in the project the more compromise he has to help and solve some issues we can have during the commissioning,so make the client active part of the project.
----------------------------------
A final in-house test with simulation software can indeed be an eye-opener for the client and deliver a great saving in time in the delivery fase (and shorten the final acceptance test). Some of these simulation software are capable of simulating the hardware I/O, which has the advantage of minimal changing of the software (preferably no changing at all), thereby reducing the risc of bugs made by the adaptions for simultion to zero.
In this way both logic, control loops (for instance PID loops) and the look and feel of the Scada can be tested and experienced.
This will lead to more understanding of the proces and the solutions by the client and the supplier (due to the comments of the client).
Together with a functional description and a solid test protocol the automation part can not go wrong in this way.
Note that the electrical installation is made according to a set of electrical drawing which also has to be checked. An active client will demand these drawing in order to check them, before the supplier may proceed with the production of the installation.
The number of installations which are delivered and accepted without a complete or error free set of drawing is too large.
Both client and supplier are capable of changing this habit.
We must conclude that delivering a succesfull automation project needs both an active client and an active supplier.
The assistance of software to make documentation and check the working can be very useful, but is not the key secret for succes.
----------------------------------
The most spectacular successes I have seen had committed champions to lead the various phases of the project who could readily accept going back to the drawing board to overcome any difficulties encountered during the execution and construction of the project
----------------------------------
In any automation, the wekest link is human being. So many new technologies /concepts/operating philospphies are evolving every day.the less we are trained/ updated, the more we fail. No woder, we should end up only in counting.
Thank you very much for the many interesting comments so far.
@Bob, I understand your concern about that the article is a little bit sales focused. However I liked this article written by my former colleague (I work in another field now) and wanted to support it. I am glad that it resulted in a discussion with many interesting and true comments that followed here in this thread.
I agree to many oppinions here that we should involve the customers as much as possible, know the technology and also mentioned inhouse tests can help increasing the projects success and having less stressful commissioning.
----------------------------------
Francis Lovering • It is interesting that this discussion has largely covered exactly the same things that have been known for years and years. Define the User requirements properly, involving the user closely, simulation testing, using modelling techniques and so on. But as I said all this has been known for years. And yet projects still go wrong.
I think that getting the right project manager and empowering them is one key. Among many other things such a manager needs enough technical skills to communicate both ways with the end users and the automation team. They are not common.
An issue that I have not seen mentioned here is one that applies to the frequent situation where an EPC engineers and buys all the equipment for the project including the automation system. Often the EPC produces the URS on behalf of the end user and sits between the user and the Systems Integrator. There are often complex interactions and sometimes conflicts between the EPC’s desire to ensure that the project is held to its cost estimates, and the end users desire to get exactly what they were hoping for.
Another major issue is Change and how to deal with it. Changes arise during a project for various reasons, often a part of the process is changed late and some parts are not completely definable until late in the project, after the system has been ordered. This is particularly true for fast track new process projects.
There are papers that address some of these points www.controldraw.co.uk/presentations/Presentations.htm
----------------------------------
I agree with Francis, but would like to add that there are 4 major items that need to be added to any project, Planning, Planning Planning and testing.
Planning with the sponsor or project initiator for the project, that he has defined what he wants to achieve.
Planning with the user, (often not the project initiator) that he gets what he wants.
And planning with the project delivery team, to ensure they develop an accurate functional spec meeting the needs of the first 2, then plan how to deliver ,review and test the system before delivery to site.
Finally Testing, commissioning the system properly, and testing the "What if" and What can go wrong scenarios not the what happens if it all works scenarios.
This takes time and costs up front, but no where like what it costs to fix later, however it depending on the environment it can be difficult to convince people of the real benefit of this approach especially in a multi disciplined environment where the overall cost means it gets little focus or understanding. However it does buy a lot of good will and acceptance of the project and its outcomes.
----------------------------------
This discussion is getting more interesting with every thread. I would also like to contribute by a background of five >million USD projects as a project manager at the user side.My assumption is; an automation project is either a reinstrumentation project or a project which is part of a bigger grass root project. I will concentrate on the first one.I think there are basically two types of customers; the ones who expects the contractor to create and submit the best project, and the ones who think that the contractor is only a tool to implement the described project. It is not usually black and white like I described above, but usually a mixture of behaviors. Second model usually helps more for successful projects, I believe.Since all the projects have two actors; i.e. the customer and the contractor, the 'agreement', and the succeeding 'communication between them' are the most critical and key elements with any project, especially in automation projects.When both parties' project managers are excited with the imagination of the project, and the agreement is good enough to handle unexpected issues, then we can say that there is a good probability of success.I will stop by a plain fact, before being boring; both parties spend a small fraction of the total intellectual effort involved in a project during the preparation phase, which will cause change orders or reluctant acceptance later. Therefore the main idea, the structure, expected performance should be clearly declared by the user, beside specifications, and database. The contractor should be transparent and specific during the bid.The following document presented in Vienna 2009 PALM Forum, is a short form of my experience on the topic. www.scribd.com/doc/43914146/Improving-Reliability-Thru-Automation-Projects-r3
----------------------------------
Haven't read all the experiences, but I had not seen many projects go wrong.
Almost always they end as expected.
Success=result-expectations.
May be lack of communication of between client/user expectations and manager/integrator objectives.
----------------------------------
We are developers of software for the manufacturing industry (mainly glass and joineries) and about 40% of our projects are factory automation. In my experience there are similarities with our other software projects in that it is almost impossible for the customer to come up with a good specification of his needs at the start. But while agile software development is a good answer to this problem in pure software projects, in factory automation projects the main costs are in the hardware and to change that part in a later stage is not easy and expensive.
However, if the customer becomes aware of his situation it is quite often easy to convince him to design a bit of flexibility into the machinery.
Furthermore we usually convince them as well to do only the basic functionality of our software first and then extend while we get experience with running the line. This has multiple advantages. First, the learning curve for the users is less steep. Second, the users can give us some input on their additional needs and will get improvements, which increases acceptance, and third it is always difficult for us to get proper interface specifications from the machinery manufacturers until shortly before they start installation, so the time window to adapt our software is usually very short, which means we prefer to deliver a smaller but properly tested subset than a final solution with lots of bugs.
----------------------------------
I completely disagree with the topic title , if it often fails I am sure automation would have been replaced. I also agree with Bob , this topic is clearly sales biased.
Automation is never a project on its own , it has many dependencies and interlocks with mechanical, pneumatic, hydraulic, IT , SCADA and people counterparts. If something is not well understood or prepared the automation guy has to adapt.
The problem is that the automation guy always comes last and gets the blame.
So yes we have to adapt and learn , projects live and need to be adjusted.
Don't get me wrong preparation is key but even then it doesn't go as expected, and then the automation guy has to write some funny logics to compensate for example the mechanical problem.
About 10 years ago something went very wrong in space because two mechanical parts did not match , the parts where made by two different companies ( all HAIL to the lowest bidder aleluhja) and they both used different tolerances. it was like one used metric and the other imperial .....so with many years and millions to prepare, enough people and skills it still goes wrong , but we learn and adapt.
----------------------------------
Francis Lovering
Not many people admit to problems with their Automation Projects, publicly at least.
And they tend to get there in the end, perhaps with less functionality than they originally hoped for and with overspends. So the project does not fail overall.
As for "and then the automation guy has to write some funny logics to compensate for example the mechanical problem.", this is a classic failing in the requirements analysis. And 'Funny Logics' are not a good thing to do.
Hans
You may think that " in factory automation projects the main costs are in the hardware", my experience is that there are many projects where this is not the case. And defining the hardware is relatively easy, it is in the software where the projects go most often wrong.
----------------------------------
Francis Lovering - 1st paragraph, BINGO! Exactly what I was trying to say, to answer the original question.
----------------------------------
I agree with you. Automation (Control and Instrumentation) is the last discipline in any project that has to make the project come to its final realisation and always suffers from project cut backs and or discounts. In my experience, automation has always suffered from budget cuts and shortcut. Other engineering disciplines always get their way when it comes to forging a 40" flange or construct a giant piping support with almost a near blank check, just because they are buying a big lump of metal, but if you try to get an extra piece of software license or, more advance control system, getting an extra 1000 dollars is like getting blood out of stone.
In a recent documentary on worldwide resources, one international recruiter stated "If you pay peanuts, you get monkeys", and worst still any monkey with keyboard claims to be control system programmer, just because they happen to learn a few graphic game codes in VB, and all of a sudden they are expert DAQ/SCADA programmers.
This has give automation a bad name in so many industries to a level where human intervention is still seen as safer bet than automatic alarm on a high pressure drill return flow line. And when the operator is fallen asleep on the job which has lead to the rupture and release of effluent into the environment inflicting 100s-1000KUSD of damages, they just record it as human error on their LTIs and lessons learned, as an acceptable practise.
----------------------------------
Francis, of course you are right, in most cases the software costs exceed those of the hardware. This simply reflects the fact that these days all the complexity is addressed in the software. What I should have said is that most providers have the software in some form already; all that’s needed is some adaption to the actual situation.
In our case we only do the overall control of the plant. That is we make sure the right pieces are produced at the right time and arrive at the right point for assembly. To this end we communicate with the machines’ PLCs which are supplied and programmed by the machinery manufacturers. They of course are in the same situation as we are. They have some form of software from previous projects which they have to adapt to the new plant.
In my experience that is where the problems come from. Quite often the customer starts building the factory before they have definitely decided on the machinery suppliers. So we know the details of the machinery and the interfaces to the PLCs only a short time before the machinery is installed. And because the PLC programmers have as little time as we have to do those program adaptions, these interfaces might even change later, because we or they discover problems in the communication.
Thus, although those program extensions usually only take a few man-months we have too many programmers involved on both sides trying to make it work in the short time given. This means a lot of communication overhead with a high potential of misunderstandings. These are then usually discovered during implementation and testing, where you are faced with other problems as well, e.g. machines not yet working properly and of course a customer that does not understand why the whole thing is not working yet.
This brings me back to the point I tried to make before. For a first version, try to implement and install the minimal functionality needed to make the plant work in the standard case. Implement additional functionality later. Usually the customer learns only after a while what all they could do with that new plant. Just recently we had a case where during specification we all agreed, that a certain configuration would never happen on that new production line. It just did not make sense to produce that there. Only half a year of running that line now the customer wanted to run exactly that and it suddenly made sense to all concerned. Fortunately it took us only a few hours to implement the software extension required to make that work.
Maybe this is a way of lowering the customer’s expectation of what they can get initially. But what they get in return are higher flexibility for changes and extensions later in the life of the plant and thus more possibilities in changing ad adapting the line to changes in demand, a changing product mix and new product to be produced over the line.
Note - the discussion is still running, and more points have been added, mostly repeating points that have already been made.
