Squirming: its Not Just for Halloween Anymore.

Remember those nameless horror films of your adolescence where three kids whip out a Ouija board for fun. With the onset of creepy music, the films protagonists eye’s widen as the planchette spells out “HE’S IN THE HOUSE AND IS GOING TO EAT YOUR BRAINS!”

If directed effectively, the audience will squirm in their seats and implore the actors to “Get the hell out of there!” Its that feeling, the one that tells you something isn’t right, and that whispers in your ear in a raspy voice “Get out while you can!”

These are the feelings I sometimes feel as I watch the Data Revolution unfold, and that I have labeled the Squirm Factor. As we unleash tools to help us derive insight and value from gargantuan data sets, I expect there to be a corollary rise in Squirm Factor amongst big data stakeholders.

In a recent blog post Edd Dumbill mentioned the Squirm Factor, and whether it could, or should, be codified. Given the vast spectrum of people involved in the Data Sphere, and individual Squirm Tolerances, it may impossible to arrive at empirical measures of “Squirmishness”.

However, we can look for visceral clues to recognize the Squirm Factor in ourselves and in others that might help us build a foundation for “data culture”, or what is and isn’t acceptable for this emerging community. Here are a few examples to start the discussion:

1. The Alien Probe Trade Off Squirm (Consumer) - aka the mortgage application. If you have been through this you’ll have the muscle memory vaguely reminiscent of being microscopically examined by faceless banking creatures. You will, however, endure the procedure, in order to obtain the funding for your new dwelling. Later hypnosis may reveal the details of your “encounter”, but do you really want to remember?

2. The Algorithm Development for Energy Trader Squirm (Data Scientist). This is squirm produced when a data scientist creates a sublime algorithm generating profit maximization trading schemes, only to hear the energy trader say “We like to create a monopoly, then induce volatility”. How can something so beautifully derived and crafted be twisted into something so desperately evil?

3. The “We’ve Tapped into Something Dark and Supernatural Squirm” (Management). This is the squirm felt by a business leader, generally an MBA, when presented with a magical analytic treasure trove by his data science Magi. However, he doesn’t understand or trust his Magi’s mysterious powers. And as a my former colleague Eric Bonabeau said, “Managers would rather live with a problem they cannot solve than accept a solution they don’t understand.”

When it comes to data, whether you are a consumer, data scientist or management, what makes YOU squirm? Go ahead, scare us all, please share you data Squirm Factors in the comments section.

Then you can return to your Ouija Board just in time for Halloween. Remember, its just a silly game. Really. Just ignore that man in the hockey mask...........

Why Data Ethos

Wikipedia defines “ethos” in part as: the guiding beliefs or ideals that characterize a community, a nation, or an ideology. In this case the community is that one emerging around “Big Data.” The community who is now collecting, mapping, reducing, and querying gigabytes , terabytes, and petabytes of data.

Our community is now breaching the realm of credit card companies (with fraud detection), Wall Street “quants” with their micro trend algorithms, and our friends in the goverment with deep pockets and Big Iron.

With open source software, commodity hardware and a few Ph.D’s we can now unlock the mysteries of huge data sets. We can now go after complex, and extremely compelling problems. And we can begin to understand, predict, and even steer human behavior to an unprecedented degree.

As interesting as the problems we chase, will be the dilemmas that emerge. Three questions and one searing comment remain etched on my mind from the early days of the revolution:

From a Casino: “Can I predict when a gambler is ABOUT to feel unlucky (so the casino could intervene to them in their casino)”

From a military researcher: “How can I increase my “Kills per Million”?

From a consumer food company: “How long can we keep trans fats in food before we risk a class action lawsuit”

and finally, from an energy trader: “We like to create monopolies, then induce volatility!”

The answers to all of the questions would (and in the case of the casino, did) deliver competitive advantage.

The comment from the energy trader preceeded the collapse of an industry and the dissolution of his company.

All of these statements also made my colleagues and me stop to think ,“Yeah, we can do this, but should we”?

Data Ethos exists to examine that spectrum from competitive advantage to bad behavior as the Big Data / Data Science community evolves.

To paraphrase Peter Parker’s (Spiderman) uncle Ben: With Big Data comes Big Responsibility!

Lets go solve cool problems! And, let’s try not the melt the earth into a soupy goo in the process!

.