AI is currently experiencing the biggest wave of investment and excitement that it has ever seen. It has become accessible and mainstream, but where did it come from? As it turns out, the development of Artificial Intelligence is a pretty natural consequence of the upward trends in the quantity of digital information, and availability of computational power. What follows is a brief exploration of these trends, and how they relate to AI.
Throughout history there have been multiple paradigm shifts in how information can spread. From cave paintings, to the invention of alphabets and languages, to the eventual creation of the printing press. Followed later by the digital age of radio, television, computers and the internet. The trend is clear. Over time we have developed newer and better ways to spread information to an ever-increasing number of people. Now, a large majority of the world’s population have access to near-infinite amounts of information.
Of course, it is the computers and the internet which are particularly relevant to this discussion, and there are a number of important points to highlight. The first of which is Moore’s law, which states that the number of transistors on a microchip doubles approximately every two years. This pattern has largely held up over the last fifty years (hence why it became known as a law). This exponential rise in compute power led to the democratisation of computers; they became smaller, cheaper and more accessible. As computers became more mainstream so did the internet, and more and more people were establishing an online presence. New digital-native services began popping up, providing convenience which was not possible otherwise. The writing was on the wall. This was the future.
Enterprises began adapting to this new online environment and quickly identified the golden goose. Personalisation. Dynamically customising the content which users see and interact with based on what generates the most revenue. There is only one issue - to accurately recommend things to users you need to know things about them. You need data. The more data you have the better decisions your systems can make. Enter Big Data - large and diverse datasets which rapidly grow over time.
Big Data comes with its own unique problems. As these datasets grow massively in size, storing them becomes a major challenge. There is also the issue of dynamic scaling. Online user traffic is not consistent. There are certain time periods where it grows exponentially over night due to external events such as holidays, product announcements, or concert tickets going on sale. Maintaining the infrastructure required to deal with these traffic spikes is extremely wasteful, as throughout most of the year it is not required. The solution to these problems is now known as cloud computing. In reality there is no “cloud”, it is just somebody else’s computer. Large corporations with spare hardware began renting it out to those who need it. As the business model proved successful, new datacenters were built specifically with this intent. This removed the responsibility of managing your own infrastructure. You no longer had to worry about storage or compute restrictions, you simply pay for what you need, when you need it. If you could afford it, compute power was effectively limitless.
This brings us to where we are today. Access to massive amounts of data coupled with the ability to process it, presented an exciting new possibility. To build systems which analyse this data, find patterns, and make decisions by themselves. Artificial Intelligence. Research in the area is receiving huge investment at the moment, and there is a lot of competition to find the next breakthrough. We are amidst a rat race. The question is - what is the cheese? Is it solving humanity’s biggest problems? Or creating an infinite stream of revenue?
A brief history of AI
While AI is the current hot topic, we need to acknowledge that in reality it is not a new concept. We must understand the history and recognise the giants whose shoulders we stand on today.
All the way back in 1950, Alan Turing published his paper “Computing Machinery and Intelligence” where he posed the question of whether machines can exhibit intelligent behaviors which are indistinguishable from humans. This paper can largely be attributed to the birth of interest in the field of AI. It sparked a plethora of research and resulted in lots of advancements in the following decades; many of which are still relevant today.
- 1951 - The first artificial neural network was built - SNARC.
- 1955 - The phrase ‘Artificial Intelligence’ was coined by John McCarthy, Marvin Minsky, Nathaniel Rochester and Claude Shannon.
- 1957 - Frank Rosenblatt developed the Perceptron, an early artificial neural network.
- 1962 - Arthur Samuel pioneered the concept of machine learning by developing a computer program that improved its performance at checkers over time. The program beat Robert Nealy in a publicised match.
- 1966 - Joseph Weizenbaum developed ELIZA. Known as the first “intelligent” chatbot despite being rule based. The capabilities of the bot raised a lot of ethical questions around human-computer interactions.
- 1969 - The concept of backpropagation was first introduced by Arthur Bryson and Yu-Chi Ho.
The 1970s, however, were not quite as promising. They brought on the first major “AI Winter” - a period of decreased funding and interest in AI research. This is largely accredited to the Lighthill report which claimed that the field had not produced the significant breakthroughs which were promised. This led to a considerable loss of support from companies and governments.
Although the excitement and funding returned in the 1980s, it did not stay for long. In 1984 at an annual meeting of the Association for the Advancement of Artificial Intelligence (AAAI) the world was cautioned of another impending AI Winter (in fact this is where the phrase was originally coined). It was predicted that history was going to repeat itself and as in the 1970s, investment and research would collapse due to inflated expectations and underwhelming results. Within just three years this became reality and the second AI Winter had arrived.
A pattern was emerging: an initial wave of hype and excitement, which in turn led to disillusionment and overblown expectations which could not be met. As we reflect on where we are today we must keep the past in mind and ride the wave of excitement with cynicism at heart, as to not repeat the same mistakes.
Nonetheless, over time the research returned and throughout the next decades many more significant advancements occurred, particularly after we entered the 21st century. Some of these are:
It is 2026 and AI has once again taken center stage. Many believe that we are in the middle of an AI Spring - but what exactly happened between 2017 and now?
Despite the title of this section, it will not discuss the Transformers movie franchise. Instead the focus will be on the transformer model architecture which reshaped the landscape of AI in the last ten years.
The idea of a model which can understand natural language and interact with a human has been around for decades - as noted above with ELIZA, Jabberwacky and many others which came along the way. However, there has been a recent revolution in the space with the arrival of the transformer architecture in 2017; introduced in the paper - Attention Is All You Need by Ashish Vaswani et al.
The transformer model is a type of neural network architecture based on attention (hence the title of the paper). This concept allows transformers to determine the relationships between each part of the input, and focus on what is most important about a specific sequence of data. It tells the AI model what to pay attention to. For example, it could interpret words within a specific context (where a word may have multiple meanings), or assess the significance of a word within a sentence. While this is very simple for humans, it was a huge problem for computers. Attention changed the game.
Consider the text below and think about how the context changes the meaning of individual words. This example has been taken from IBM’s page about transformers.
“on Friday, the judge issued a sentence.”
- The preceding word “the” suggests that “judge” is acting as a noun. As in, a person presiding over a legal trial rather than a verb meaning to appraise or form an opinion.
- That context for the word “judge” suggests that “sentence” probably refers to a legal penalty, rather than a grammatical “sentence.”
- The word “issued” further implies that “sentence” refers to the legal concept, not the grammatical concept.
- Therefore, when interpreting the word “sentence”, the model should pay close attention to “judge” and “issued”. It should also pay some attention to the word “the”. It can more or less ignore the other words.
Historically, neural networks ingested input data sequentially, one at a time, and in a particular order. This approach faced issues with performance and keeping track of relationships between distant data points, particularly with large inputs. Attention mechanisms do not face these limitations. Not only do they process the entire sequence simultaneously (improving the ability to understand long-range dependencies), but this quality also enables the computations to be done in parallel rather than one at a time. This allows transformers to take full advantage of the capabilities of the GPU (which greatly outperforms the CPU in parallel tasks), and solve many of the performance issues faced in older neural network solutions.
One of the first majorly successful applications of transformers was BERT - Bidirectional Encoder Representations from Transformers - introduced by Google in 2019. It could predict and classify input text. The model quickly became ubiquitous, and is still used today as the basis of Google search. Despite its fame though, BERT was soon overtaken in popularity by OpenAI’s GPT models.
GPT stands for Generative Pre-trained Transformer. GPTs generate new data by applying the patterns they identified in pretraining as a response to user input. In short, they are fed unlabeled data and forced to make sense of it on their own. After the pretraining stage, the model can be fine-tuned towards a specific task with the use of labeled data. Fine-tuning can also be combined with a ‘reinforcement learning from human feedback’-based objective. By finding patterns in these datasets, GPTs can then draw similar conclusions when exposed to new inputs such as a user’s prompt, and generate a response.
With the release of ChatGPT in 2022, the general public was introduced into the world of Generative AI, which has taken the world by storm. AI has never been so mainstream and accessible before. This begs the question - are we ready for it?
