https://www2.econ.iastate.edu/tesfatsi/DeepLearningInNeuralNetworksOverview.JSchmidhuber2015.pdf

Deep learning in neural networks

Prelude. Cycle 1. Introduction. Cycle 2. Structure defines function. Cycle 3. Diversity of cortical functions is provided by inhibition. Cycle 4. Windows on the brain. Cycle 5. A system of rhythms: from simple to complex dynamics. Cycle 6. Synchronization by oscillation. Cycle 7. The brain's default state: self-organized oscillations in rest and sleep. Cycle 8. Perturbation of the default patterns by experience. Cycle 9. The gamma buzz: gluing by oscillations in the waking brain. Cycle 10. Perceptions and actions are brain state-dependent. Cycle 11. Oscillations in the "other cortex:" navigation in real and memory space. Cycle 12. Coupling of systems by oscillations. Cycle 13. The tough problem. References.

Rhythms of the brain

‡§ ¶ Humanity has just crossed a major landmark in its history with the majority of people now living in cities. Cities have long been known to be society’s predominant engine of innovation and wealth creation, yet they are also its main source of crime, pollution, and disease. The inexorable trend toward urbanization worldwide presents an urgent challenge for developing a predictive, quantitative theory of urban organization and sustainable development. Here we present empirical evidence indicating that the processes relating urbanization to economic development and knowledge creation are very general, being shared by all cities belonging to the same urban system and sustained across different nations and times. Many diverse properties of cities from patent production and personal income to electrical cable length are shown to be power law functions of population size with scaling exponents, , that fall into distinct universality classes. Quantities reflecting wealth creation and innovation have 1.2 >1 (increasing returns), whereas those accounting for infrastructure display 0.8 <1 (economies of scale). We predict that the pace of social life in the city increases with population size, in quantitative agreement with data, and we discuss how cities are similar to, and differ from, biological organisms, for which <1. Finally, we explore possible consequences of these scaling relations by deriving growth equations, which quantify the dramatic difference between growth fueled by innovation versus that driven by economies of scale. This difference suggests that, as population grows, major innovation cycles must be generated at a continually accelerating rate to sustain growth and avoid stagnation or collapse. population sustainability urban studies increasing returns economics of scale

/pdf/growth-innovation-scaling-and-the-pace-of-life-in-cities-4g26tvit5r.pdf

Growth, innovation, scaling, and the pace of life in cities

Cell phones and other portable devices are equipped with a variety of technologies by which existing functionality can be improved, and new functionality can be provided. Some relate to visual search capabilities, and determining appropriate actions responsive to different image inputs. Others relate to processing of image data. Still others concern metadata generation, processing, and representation. Yet others relate to coping with fixed focus limitations of cell phone cameras, e.g., in reading digital watermark data. Still others concern user interface improvements. A great number of other features and arrangements are also detailed.

Methods and Systems for Content Processing

Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.

Training language models to follow instructions with human feedback

A radical and optimistic view of the future course of human development from Ray Kurzweil, whom Bill Gates calls "the best person I know at predicting the future of artificial intelligence."

/pdf/the-singularity-is-near-when-humans-transcend-biology-2mzy4i340l.pdf

The Singularity Is Near: When Humans Transcend Biology

In this chapter, Ray Kurzweil presents and defends his view that we will reach a technological singularity in the next few decades, which he defines as a “period during which the pace of technological change will be so rapid, its impact so deep, that human life will be irreversibly transformed.” Kurwzweil argues that the pace of technological change, particularly with respect to information technologies, is exponential, and that we are near the “knee” of the exponential curve (i.e. the point at which the curve changes from largely horizontal to largely vertical). Kurzweil predicts that a core feature of the singularity will be the merging of biological and machine intelligence, such that the majority of “human” intelligence will become non-biological, and the merging of virtual and physical reality. Kurzweil considers this the next step in human-machine co-evolution.

The Singularity Is Near

From the Publisher:
Ray Kurzweil, called a "restless genius" by The Wall Street Journal, is responsible for some of the most compelling technology of our era. The brains behind the Kurzweil Reading Machine (which helps Stevie Wonder read his mail), the Kurzweil synthesizer, and the voice-recognition program that appears on Windows 98, he is also a formidable thinker who a decade ago predicted the emergence of the World Wide Web and that a computer would beat the world chess champion. Finally, someone with the authority to speak about the future also has the courage and imagination to do so. The Age of Spiritual Machines is no list of predictions but a framework for envisioning the 21st century in which one advance or invention leads inexorably to another. After establishing that technology is growing exponentially, Kurzweil forecasts that computers will exceed the memory capacity and computing speed of the human brain by 2020, with the other attributes of human intelligence not far behind. By that time paraplegics will be able to walk by using a combination of nerve stimulation and robotic devices. You will be able to choose the personality of your automated computer assistant, who will conduct business on your behalf with other automated personalities. A mere nine years later, you will be able to enhance your intelligence with neural implants. The upshot is that human identity will be called into question as never before, as a billion years of evolution are superseded in a mere hundred by machine technology that we ourselves have created. We will become cyborgs, but what will computers become?

The age of spiritual machines: when computers exceed human intelligence

The Age of Spiritual Machines

We present LaMDA: Language Models for Dialog Applications. LaMDA is a family of Transformer-based neural language models specialized for dialog, which have up to 137B parameters and are pre-trained on 1.56T words of public dialog data and web text. While model scaling alone can improve quality, it shows less improvements on safety and factual grounding. We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding. The first challenge, safety, involves ensuring that the model's responses are consistent with a set of human values, such as preventing harmful suggestions and unfair bias. We quantify safety using a metric based on an illustrative set of human values, and we find that filtering candidate responses using a LaMDA classifier fine-tuned with a small amount of crowdworker-annotated data offers a promising approach to improving model safety. The second challenge, factual grounding, involves enabling the model to consult external knowledge sources, such as an information retrieval system, a language translator, and a calculator. We quantify factuality using a groundedness metric, and we find that our approach enables the model to generate responses grounded in known sources, rather than responses that merely sound plausible. Finally, we explore the use of LaMDA in the domains of education and content recommendations, and analyze their helpfulness and role consistency.

Raymond C. Kurzweil

Papers

The Singularity Is Near: When Humans Transcend Biology

The Singularity Is Near

The age of spiritual machines: when computers exceed human intelligence

The Age of Spiritual Machines

LaMDA: Language Models for Dialog Applications