A problem with technology that collects ambient data is that it is basically spying on everything all the time. This creates two distinct issues.

1) The first very obvious problem is with privacy. I say this is obvious even though Google just claimed they made a “mistake” collecting all kinds of wireless data around the world with mobile sniffers.

Regulation through policies and procedures is usually proposed as the solution to this first problem. The fact that Google is being threatened with legal action by privacy officials in numerous countries is an example of how this control point can work. Technology also can help with authentication, authorization and live audit trails of who accessed what data and when.

2) The second problem is that too much data will overwhelm analysts, and analysts are expensive. Collecting too much data is not only bad for PR and legal conformance, it also makes a surveillance system impossible to keep up with and make useful. Who has the time or resources to keep up with massive amounts of information and find anomalies quickly? Automation technology is typically proposed as the solution to this problem, but it can also be expensive.

Que the military. They can justify the cost of solving the second problem. The military operates in environments where they collect massive amounts of data unfamiliar to most analysts (training becomes more specialized so costs are far higher) and time to respond is more of the essence. It also helps them that the first problem quickly erodes when dealing with data in a hostile environment.

Take this example provided by BBC News

One technology that BAE Systems trialled, known as a “hyperspectral camera”, is able to analyse colour – to distinguish a camouflaged vehicle from the vegetation it is concealed within.

Gary Bishop from BAE’s Advanced Technology Centre in Bristol told BBC News: “You see things with your eyes in three wavelenths, the hyperspectral camera gives you information in 10.”

The system measures each wavelength of light being reflected by an object – it can see the specific type of green that is produced by chlorophyll in plants, and distinguish that from the green of paint or dye.

Everything in the article centers around systems that create data mining efficiencies.

The military needs to quickly detect unusual patterns within otherwise normal data. This, as mentioned above, is good not only for automation but it also has the secondary effect of helping to protect privacy in civilian surveillance systems.

Automation means humans can be removed from the role of sifting through private and protected information to find a suspicious data point. The surveillance system could be setup to keep everything under wraps and only expose information required to review and confirm. Access only to suspicious event data is far less controversial than access to all data. The more limited access also can be logged and audited later. That means in the end you get access to more data but less privacy risk…assuming you trust and verify the system is operating properly.

This still begs the question of whether it is ethical to collect data in the first place, as in the case with Google. What were they thinking?

“If the company is fighting this so hard, it suggests there is more to this than meets the eye,” said Mr. Davies, of Privacy International. “The real question is: What was Google collecting from unwitting individuals and why? So far, nobody really knows.”

Perhaps at this point they should try to mount a “we were trying to find terrorists” defense…certainly sounds better than “programming error” that ran around the world and for an extended time gathering a massive amount of data.

I have to wonder for a company that has a very public emphasis on hiring the best and brightest whether they really expect anyone to believe that surveillance was not intentional. Most security professionals balk at the idea of capturing packets from random airspace — it’s known to be unethical if not illegal in most contexts. Why Google did not properly account for the risks of surveillance is hard to understand.