“Progress and Challenges of Deep Learning and AI”

It's important to start by clarifying what deep learning is good at and why we think it works so well. While conventional machine learning was able to detect patterns in proportion to feature space, thanks to a distributed representation, deep learning can innovatively model a strikingly large amount complex patters.

Deep learning has the advantage of generalizing new patterns not included in the training data based on a large quantity of computational resources and data sets labeled by human beings. This has already been implemented by using the principle of compositionality, which transmits pre-existing knowledge of the real world. The advantage of compositionality includes model-driven architectures.

Compositionality is the principle by which complex concepts are composed from simpler individual components: this appears to be an almost universal principle of nature and naturally produced data. For instance, in human language, complex concepts are represented in sentences by composing simpler individual words. In natural images, a scene is composed of sub-parts containing a multitude of individual objects and objects are composed of parts. Convolutional Neural Networks successfully exploit this structure by processing images using layers of local connections, convolution.

Thanks to fundamental breakthroughs deriving from compositionality as well as many technological advances, large-scale datasets and computational resources, deep learning has achieved impressive results in several fields, including speech recognition, computer vision, and machine translation.

Recently, it has been proven that the combination of deep learning and reinforcement learning works well in the fields of strategy and planning, as well as for control.

For example, Google DeepMind AlphaGo has successfully defeated a world champion Go player, which was a long-time grand challenge for AI. NVIDIA demonstrated driving its self-driving car based on a controller that has learnt a large quantity of image data. These examples showed us that deep learning has been developed enough to understand the meaning of inputs and to reflect it into an appropriate model.

Naturally, the next AI grand challenge will be to generate human-level “common sense”. Learning human-level common sense is equivalent to artificial general intelligence (AGI), which was generally considered to be impossible with previous AI. Its initial achievement was automated video caption generation, which translates a distributed representation obtained by a device into human language. This shows the potential of autonomous acquisition of common sense, at least under the points to a path where improvements in unsupervised learning and in language understanding may one day allow for autonomous acquisition of common sense, at least under restricted circumstances, by progressing learning without teachers and comprehension of word meanings.

First of all, there are a variety of approaches to become successful AI research organizations, including DeepMind acquired by Google and Facebook AI research (FAIR).
The structure of successful AI R&D organizations is rather peculiar: different successful approaches emerged, including Google’s DeepMind acquisition, Facebook AI Research (FAIR) established by a team of the same university, and OpenAI, a non-profit AI research company.

Similarities among these organizations include the fostering of stable research teams, open collaboration, and to focus on challenging long-term missions. To be successful in a competitive environment like AI today, it is essential to ensure a stable operational structure for mid-to-long term.

On the next stage, to pass down inventions to products, it is important to foster proactive interactions between research and business units while avoiding pressure from short-term budget acquisition and solid-line reporting. Frequent interactions between engineers and scientists are created by a co-location environment, where R&D and product units work in the same facility, not by slow and complicated solid-line reporting.

The second factor relates to how research is conducted: world-class AI research projects are carried out in cooperation with numerous research institutions, at both companies and universities. Therefore, publishing early-stage source code and their achievements to the public becomes a key factor for AI research. To protect research ideas, publishing research achievements and source code on web-based open-source library such as arXiv and GitHub in the early stage results in great cost performance alternative to patents. Keeping precious research into a desk drawer won’t bring any benefits to companies, because the developed technology based on this research won’t be adopted and its development progress will stop. Publishing information and source code proactively is a key element for today rapid AI progresses. No matter how hard AI researchers work, they can’t be successful without contribution to open-source ecosystems.