Trading analysis has become more important for savvy investors than instinct.
Instinct is a pattern of different behaviours most creatures are born with.
Business instinct, however, is something acquired over many years of observing deals, rivals and numerous other phenomena that tell a trader when the timing is right to buy or sell.
But the young pretenders aren't interested in instinct. They haven't the time to build up this level of knowledge through experience.
Instead, they are using analysis to come up with the trading patterns they need to make a killing on the stock exchanges.
Trading analysis is integral to high-frequency and other program trading techniques and requires three key elements.
- Data. In this new age of "big data", masses of information is available on all kinds of subjects, gathered from our daily activities such as shopping, using the internet, going on holiday, driving – data can be gathered from just about every conceivable thing we do
- Logic. We use logic to best interpret outcomes, and by combining data and logic you end up with a program
- Processing. To run the kind of programs used in trading, lots of processing power is needed. We use computers to process the data and logic into an algorithm that can detect various patterns and profit on them
We'll return to logic shortly when examining its development in computer trading, but this article is mainly about data and its growing role in how we make sense of the modern world.
Origins of data analysis
Imagine you've just conquered a new country. Warfare doesn't come cheap and armies cost a lot to feed.
To the victor, the spoils. But this country is new to you. Exactly how rich has it made you?
William the Conqueror embarked on one of history's most ambitious data-gathering projects a few years after his conquest of England in 1066, and the resulting Domesday Book was completed in 1086.
It detailed population, use and ownership of land, livestock and other resources that could be taxed to boost the Crown's coffers and stolen then offered to friends, and enemies, to keep them sweet.
The Domesday Book was a landmark in the use of data.
Data analysis refined
Florence Nightingale is best remembered in history as the "lady of the lamp" for her tireless ministerings of the wounded and sick when a nurse during the Crimean War.
She soon found, however, that far more soldiers were dying from preventable diseases and post-surgery infections, than were actually dying as a direct result of their wounds.
Taking her findings to the War Office in 1856, she needed a striking way of visualising this data for the gathered ministers.
Her inventive use of "coxcombs" (below) – a form of pie chart now known as a polar area diagram – accurately depicted the worst months of the war and the gradual improvement as her measures to sanitise hospital recovery wards took effect.
The data provided Parliament and the medical profession a stark visualisation of human lives as numbers.
Modern statistical analysis and the programming behind many trading analysis algorithms owe their existence to a Lincoln-born Victorian mathematician called George Boole.
Boole introduced a form of algebra where, instead of denoting numerical values his algebraic expressions denoted truth values – either true or false – that are represented by the binary digits 0 and 1.
These binary digits are then used to test a series of propositions using any combinations of the three functions: not, and, or. These functions are called "logic gates".
Through work done since, such as that carried out by the code-breakers at Bletchley led by Alan Turing, to today, where all modern computers perform their functions using binary logic, all owe a debt of gratitude to Boolean algebra.
The path to "Big Data"
The most significant progress in data analytics has been in the last decade or so, however.
Cloud computing has established a new benchmark in processing, the like of which was previously only available to those that had access to massive data and storage centres that could cost millions of pounds.
And George Boole would admire the circularity of logic that now puts the computer – previously just a tool that uses logic to process data – at the cutting edge of data gathering itself.
Every bank transaction you make; every online purchase – the price you paid, how long it took to arrive; each time you use a store discount or points card; your telephone history; the internet pages you visit, all this and much, much more is gathered through computer usage and analysed every day to spot patterns in your life.
These patterns can be used by advertisers to better target products they can sell to you. These patterns can also be used by data analysts who work with the police to spot odd behaviour that may be linked to criminal or terrorist activities.
And as the cloud opens up vast new processing capabilities at a fraction of the cost of running your own data centre, masses of binary data from financial trading can be analysed in seconds to form the basis of algorithms for program trading.
As the data gets bigger, so do the ideas.
In just the last four or five years, great strides have been made in a form of artificial intelligence called "machine learning".
Data can be processed and unfavourable outcomes identified and corrected as computers "learn" to spot anomalies and deal with them.
Trading analysis has now made it possible for computers to identify trading patterns through a process called "data mining".
Your computer algorithm can test various trading strategies by mining historical price and trading data for examples of favourable outcomes so it can predict similar conditions that are likely to produce similar favourable outcomes.
The future is data
Just as a manager relies on his personal assistant to take care of simple, but time-consuming administrative tasks, business is coming to rely more on technology to solve similar problems.
Financial technology – or fintech – is a relatively new business sector that is developing new cloud-based solutions that are able to handle an ever-growing deluge of data.
These solutions are not just developed for the sole pursuit of trading, but also in the management of post-trade activities.
The global banking industry is placing increasing levels of trust in third-party fintech firms that offer such services as liquidity and collateral management, regulatory reporting, post-trade analysis and other services.
Not only do these services analyse much more data than the banking industry’s legacy IT systems are even capable of safely storing, they are saving the industry money by identifying bad practices and poor trades.