“Beyond Data” in the Intelligence Sector
Bob Gleichau from In-Q-Tel wrote an interesting article entitled “Beyond Data.” As the article discusses, the intelligence sector adds complexity onto the difficult job of sorting, searching, and understanding large data sets. Some of the challenges Bob wrote about include:
- The security clearance level maze between dozens of intelligence, homeland security, and financial regulatory agencies
- Laws against collecting data on US citizens in certain cases, for example biometrics
- Legacy government systems
There are also many additional challenges:
- Disconnected security networks
- Low network bandwidth and high latency in overseas locations, particularly war zones
- Paper (no joke- in 2011 paper still holds a vast amount of the government’s data)
- The ones who pay the checks are not system end users
- Government contract structures favor waterfall over iterative development processes
- No access to commodity cloud infrastructure services such as Google App Engine, Rackspace, or Amazon Web Services to perform distributed data analysis
One of the most interesting ideas from the article was embedding great developers with intelligence analysts to create and execute very complicated queries. I’m sure some agencies are doing this already but from my experience it’s not a common practice. (Private industry may need to do this in the future as well but that’s a different topic.)
A second important idea is the concept of allowing full search capabilities but masking search output when a user’s clearance is not high enough to see results. This is a very hard problem that involves user authority and access management, metadata mark up, and clear, unambiguous rules for clearance resolution.
Finally, one last concept that isn’t in the article but is crucial. The government needs to be careful of throwing money at hard problems. Building information systems (including data analysis systems) isn’t like designing a new fighter jet. It’s amazing what a small team of six to eight capable software developers with a passion for intelligence community domain challenges can accomplish when given access to large data sets and the freedom to choose their own tools. That’s why companies like LinkedIn, Facebook, and Google are successful with using data to generate business value.
Article: Beyond Data (IQT Quarterly), see also Data Science in the U.S. Intelligence Community (IQT Quarterly)
