The Distributed Data Warehouse

It seems the company behind the FBI’s cutting-edge data search engine wants to provide the same services to the private sector. Chiliad, with corporate headquarters in Herndon Virginia, has been providing a distributed data search engine for the FBI ‘s counter terrorism datawarehouse for many years now. This distributed and parallel processing architecture was conceived primarily to get around the need for a central storage area/site and the need to gather data from disparate sources; much like the key data warehousing issue currently facing many organizations. Another benefit of this technology that organizations could reap is the ability to more easily and effectively ‘connect-the-dots’ with an ever-increasing amount of structured and un-structured corporate data .

With their architecture, no centralized storage is necessary. They have what are called Discovery/Alert nodes which are placed wherever information is managed and are part of a secure peer-to-peer network which allows a query to be processed in parallel, decreasing the amount of time needed to get a result.

It should be interesting to see how they will market to businesses. Off-hand I’m thinking they will pitch to corporations that are close to the same size as the FBI in terms of data, data sources, complexity, etc… This way they can pretty much turn-key their existing production solution. After getting some big players on board and replenishing their coffers, I think they will then look at scaling for a wider variety and size of clients.

I wonder why it took ~6 years for them to make the foray into the corporate world. I’m thinking it may have been an exclusive deal with the government, but maybe they wanted a solid, long term, and respectable government client first to increase credibility and ease their marketing efforts? Anyone have any insight here?