This is an interesting time to be a player in the data integration space. The past few years have witnessed the maturation of NoSQL databases as popular data stores, which evidences the explosion of types of data that organizations need to capture, store and analyze. For many years, the bulk of data business dealt with was structured data produced by ERP, CRM and related systems, and the integration driver was data warehousing for business intelligence purposes.
Fortunately, the tooling is evolving together with the challenge. For instance, EC Wise uses the integration toolset in Talend Studio, which includes connectors for NoSQL databases like Cassandra, Neo4J, HBase, MongoDB, CouchDB, Couchbase, and Riak. These connectors allow EC Wise integration specialists to work with these databases, as well as with SQL databases, queues and applications through the common abstractions of data maps and flows. In addition to support for NoSQL databases, vendors are adding support for stream processing and meta data management which were historically very specialized and costly niche capabilities.
At the high level, EC Wise undertakes two main classes of data integration projects. In one class, processes assemble and often aggregate data for analytic purposes, whether in a data warehouse to facilitate Business Intelligence reporting or in a data lake to enable various types of analytic initiatives. In the other, data from multiple sources needs to be brought together in a common form for further processing. Applications that EC Wise developed exemplifying this class include preparation of customer statements for presentment via paper or e-bill, health insurance enrollment, and marketing systems.
At the end of the day, a data integration project is a type of software development project. That becomes obvious when you realize that the end result of using a graphical editor to model data transformations and processes is a set of java packages that the Talend engine executes or a set of .NET classes that Microsoft SQL Server Integration Server executes. As a company that develops complex software systems for its clients, EC Wise can apply software engineering techniques and practices to these projects. Such techniques include logging strategies, test driven development and secure programming. In addition, the EC Wise team’s deep expertise in a variety of RDBMS and NoSQL databases enables it to design performance optimizing ETL transformations.