I started my career as an A>I (Ab Initio) developer. Worked on it for 3.5 years before moving to Apache Spark with Scala. A>I as a tool is not as common as Apache Spark, but that is mostly because the licensing cost of A>I is very high. So small or medium sized companies cannot afford it. It's the big traditional multinational companies (Banks, Telecom etc) that uses A>I. And in these companies A>I is used more than Spark. The reason A>I is preferred by these companies is because it is very reliable and performative. You don't need to fine tune jobs based on your cluster configuration (like you do in Apache Spark) to make it work. You just need to write optimised code, but that you need to do with Apache Spark as well. A>I just works and in most cases it will give as good a performance as Apache Spark. Yes, it's that good in performance. A>I support is also one of the best I have seen, which is very important for big companies. If you have any queries related to your A>I job performance or looking for support to some complex issues you are facing, the support team is very helpful. It's easy to learn if you have access to A>I tool. The connectors makes it very easy to visualise how the data will flow and transform. The A>I help file (documentation) is one of the best I have ever seen. That help file is the bible for learning anything you need to know about A>I. Having said all that, the reason I moved from A>I to Apache Spark was because I was interested more in writing code than using a GUI. My educational background is not in coding, and I was interested to get into it. It's not Also like the other comment said, A>I is only used by big companies. So I wanted to learn something new which can open doors to more companies as well. More on reddit.com

I worked with Ab Initio products for many years, and was considered an expert until I got the startup bug and left the enterprise world three or four years ago. Their core offerings are a data integration platform, called the Co>Operating System (Co>Op), and a graphical interface for developing applications that run on that platform, appropriately called the Graphical Development Environment (GDE). They provide a whole suite of software that works together with these products to tackle everything from metadata management to job orchestration to business interface presentation. A sample Ab Initio job would be something like this: you have a flat file from a mainframe that's partitioned by a key and stored on a network filesystem, and you want to join it to a traditional database table on a common key. You then want to take a few columns from each source and combine them into an output flow, then transform that flow to add derived columns, re-sort the data, change the key, aggregate some values, etc. The end result should be written to a columnar store to be used for analytics. You would start by defining the database connections and the layouts for the different sources and targets using Ab Initio's proprietary markup language. You would then build a new "graph" in the GDE with components to unload the database table, read in the partitioned file, prepare the flows to be joined (this usually means making sure the inputs are partitioned and sorted by equivalent keys) and join them, perform any other transformations, and load the columnar table. You literally draw lines between the components to indicate the data flows, and you can pop open each component to adjust its behavior. You then deploy this graph to a beefy system running the Co>Op, and schedule it to run e.g. every day—triggered when the mainframe file is updated—to keep the analytics table fresh. And of course that's just the tip of the iceberg. You can define graphs that run continuously, interact with web services or present their own endpoints, implement complex business logic, call remote procedures, etc., and all the metadata about the sources, targets, and the graph itself gets loaded into a separate datastore to be reviewed, audited, and otherwise analyzed. One company I worked for had literally thousands of graphs and tens of thousands of jobs (graph executions) running on an absolutely enormous cluster with hundreds of hardware threads, terabytes of RAM, petabytes of SAN, etc. The Co>Op makes it a breeze to run shared-nothing jobs distributed over multiple servers, with each partition of each component living in its own process, and data being retrieved and retransmitted between processes and over the network as needed. If you're thinking, "Hey, that sounds a lot like Map/Reduce," you're right! You can think of Ab Initio's platform and its ilk as precursors to Hadoop and other "big data" platforms, with a lot more polish (at the time, at least). In fact, when I left the scene they were working on making Ab Initio run on Hadoop clusters and read from and write to HDFS. All this comes at a cost, of course, and Ab Initio has the most expensive data integration platform by a large margin. They cater to the biggest of big businesses and charge an arm and a leg for their software and services. The Co>Op's features are individually licensed by how many integer operations your processors can perform per unit time (SPECint). I have heard rumblings of pushback from the companies where I used to work; there's interest in diversifying away from Ab Initio, primarily for cost reasons. Ab Initio has failed to keep their licensing model updated for the cloud era. Unfortunately, nothing is quite as good as Ab Initio's software, and there are huge retraining/restaffing/rewriting hurdles there. Ab Initio's main competitors back in the day were Informatica, DataStage, SAS, Pentaho, etc., with Talend being the only open source option. Nowadays you have about a million different options like NiFi and Spark. I've been out of the picture for so long that it's hard for me to say definitively what the landscape looks like. I feel confident saying that Ab Initio will be around for a long time, like COBOL, but I don't know how much longer their software products can stay first-class citizens in the Fortune 500 businesses that license them. Okay, let me try to answer your actual question: What's their pay, work, reputation like? It's a great company. I would compare it to Google. They are engineers par excellence. When you email their support team to ask a question, there's a pretty good chance that someone with a PhD in e.g. physics will answer you. Richard Feynman's son works there. I highly recommend reading about Thinking Machines to get a feel for the culture. It's some pretty lofty shit. You will learn a lot, and I'm envious of anyone who gets an opportunity to work there. I don't have any concrete information about their compensation structure, but it's not hard to imagine that they pay their engineers quite well, based solely on what they charge for their software. If you're willing to travel as a field engineer it would probably be very lucrative. There are few specialties for an independent contractor that can command a higher rate, yet their employees never leave. (Well, almost never.) That should tell you something. Yes, they are secretive, especially if you haven't signed their NDA. They highly value their own intellectual property and the trade secrets of their customers. It's weird in this day and age to not be able to google a company and learn everything about it and the technology they make, including reference manuals, etc., but that's just how they be. EDIT: grammar More on reddit.com