Inferred Modular Superparrots

The buzz about ChatGPT and related efforts has been surprisingly resistant to the standard deflationary pressure of the Gartner hype cycle. Quantum computing definitely fizzled but appears to be moving towards the plateau of productivity with recent expansions of the number of practical qubits available by IBM and Origin in China, as well as additional government funding out of national security interests and fears. But ChatGPT attracted more sustained attention because people can play with it easily without needing to understand something like Shor’s algorithm for factoring integers. Instead, you just feed it a prompt and are amazed that it writes so well. And related image generators are delightful (as above) and may represent a true displacement of creative professionals even at this early stage, with video hallucinators evolving rapidly too.

But are Large Language Models (LLMs) like ChatGPT doing much more than stitching together recorded fragments of texts ingested from an internet-scale corpus of text? Are they inferring patterns that are in any way beyond just being stochastic parrots? And why would scaling up a system result in qualitative new capabilities, if there are any at all?

Some new work covered in Quanta Magazine has some intriguing suggestions that there is a bit more going on in LLMs, although the subtitle contains the word “understanding” that I think is premature. At heart is the idea that as networks scale up given ordering rules that are not highly uniform or correlated they tend to break up into collections of subnetworks that are distinct (substitute “graphs” for networks if you are a specialist). The theory, then, is that the ingest of sufficient magnitudes of text into a sufficiently large network and the error-minimization involved in tuning that network to match output to input also segregates groupings that the Quanta author and researchers at Princeton and DeepMind refer to as skills.… Read the rest

B37-20047: Notes / Personal / Insights

NOTE: 250-word flash fiction for my critique group, Winter Mist, at Willamette Writers

I’m beginning to suspect that ILuLuMa is not who she claims to be. Her messages have become odd lately, and the pacing is off as well. I know, I know, my job is to just respond from my secure facility, not worry about the who or why of what I receive. It’s weird we’ve never met, though. The country is not at risk as far as I can tell from the requests, but I still hold, without a whiff of irony, that the work I do must be critical for someone or something.

Still, the requests for variants of mathematical proofs set to music or, more bizarrely, Shakespearean-voiced tales of AI evolution, don’t have the existential heft of, say, wicked new spacecraft designs or bio-composite materials. What is she after? I started adding humorous little asides to some of my output, like my very meta suggestion that Hamlet failed to think outside the Chinese Room. Crickets every time. But maybe I’m thinking about this the wrong way. What if ILuLuMa is just an AI or something programmed to test me or compete with my work at some level? That would be rich, an AI adversary trying to learn from a Chinese Room. Searle would swirl. I should send her that. Rich.

Oh, here’s one now: “Upgrade and patch protocol: dump to cloud bucket B37-20048 and shut down.” Well, that sounds urgent. I usually just comply at moments like this, but maybe I’ll let her sweat a bit this time.… Read the rest

Entanglements: Collected Short Works

Now available in Kindle, softcover, and hardcover versions, Entanglements assembles a decade of short works by author, scientist, entrepreneur, and inventor Mark William Davis.

The fiction includes an intimate experimental triptych on the evolution of sexual identities. A genre-defying poetic meditation on creativity and environmental holocaust competes with conventional science fiction about quantum consciousness and virtual worlds. A postmodern interrogation of the intersection of storytelling and film rounds out the collected works as a counterpoint to an introductory dive into the ethics of altruism.

The nonfiction is divided into topics ranging from literary theory to philosophical concerns of religion, science, and artificial intelligence. Legal theories are magnified to examine the meaning of liberty and autonomy. A qualitative mathematics of free will is developed over the course of two essays and contextualized as part of the algorithm of evolution. What meaning really amounts to is always a central concern, whether discussing politics, culture, or ideas.

The works show the author’s own evolution in his thinking of our entanglement with reality as driven by underlying metaphors that transect science, reason, and society. For Davis, metaphors and the constellations of words that help frame them are the raw materials of thought, and their evolution and refinement is the central narrative of our growth as individuals in a webwork of societies and systems.

Entanglements is for readers who are in love with ideas and the networks of language that support and enervate them. It is a metalinguistic swim along a polychromatic reef of thought where fiction and nonfictional analysis coexist like coral and fish in a greater ecosystem.

Mark William Davis is the author of three dozen scientific papers and patents in cognitive science, search, machine translation, and even the structure of art.… Read the rest

Sentience is Physical, Part 2

Having recently moved to downtown Portland within spitting distance of Powell’s Books, I had to wander through the bookstore despite my preference for digital books these days. Digital books are easily transported, can be instantly purchased, and can be effortlessly carried in bulk. More, apps like Kindle Reader synchronize across platforms allowing me to read wherever and whenever I want without interruption. But is there a discovery feature to the shopping experience that is missing in the digital universe? I had to find out and hit the poetry and Western Philosophy sections at Powell’s as an experiment. And I did end up with new discoveries that I took home in physical form (I see it as rude to shop brick-and-mortar and then order via Amazon/Kindle), including a Borges poetry compilation and an unexpected little volume, The Body in the Mind, from 1987 by the then-head of University of Oregon’s philosophy department, Mark Johnson.

A physical book seemed apropos of the topic of the second book that focuses on the role of our physical bodies and experiences as central to the construction of meaning. Did our physical evolution and the associated requirements for survival also translate into a shaping of how our minds work? Psychologists and biologists would be surprised that there is any puzzlement over this likelihood, but Johnson is working against the backdrop of analytical philosophy that puts propositional structure as the backbone of linguistic productions and the reasoning that drives them. Mind is disconnected from body in this tradition, and subjects like metaphors are often considered “noncognitive,” which is the negation of something like “reasoned through propositional logic.”

But how do we convert these varied metaphorical concepts derived from physicality into something structured that we can reason about using effective procedures?… Read the rest

A Mammoth Undertaking: Interviewing the Future of Art

The following is my interview of DeepAI and Stable Diffusion running on an AMD Ryzen with NVIDIA GeForce RTX 3090.

Thank you for taking the time to discuss this important and interesting topic. Why are mammoths so important to the art world?

That the mammoths were domesticated before the start of the human era seems implausible from a historical perspective. In fact the majority of mammoths that people find in museums, libraries, and other collections are not domesticated mammoths. This suggests there might have been a greater need for mammoths within culture than before. This would make sense, considering that Neanderthals likely hunted other types of other species before settling in our world more than 400 years ago. Because they were not domesticated, they were only adapted to hunting rodents and other plant and animal species that were less domesticated in their wild form such as rabbits, snakes, etc.

Starting at the earliest part of your collection, the mammoth is an unusual theme for Rembrandt, isn’t it?

It’s hard to believe that after all the success of their latest work, The Lord’s Prayer doesn’t want to let their “biggest hit yet”, the classic The Lord’s Prayer, disappear. The actor’s latest film, The Wrestler (via Warner Bros., which produces on their own, and in the U.K.), has garnered more than 15 million views on Amazon Instant Video (the company’s preferred store for Netflix-wielding subscribers, not the usual suspects Google Video), and has become so popular that when the film premiered at the Venice Film Festival, it was deemed the top film at the fest. The star-studded line-up has included the likes of Peter Lorre, Michael J Freberg, Colin Farrell, and Peter Sellers (both of whom made it to a Hollywood winery).… Read the rest

Sentience is Physical

Sentience is all the rage these days. With large language models (LLMs) based on deep learning neural networks, question-answering behavior of these systems takes on curious approximations to talking with a smart person. Recently a member of Google’s AI team was fired after declaring one of their systems sentient. His offense? Violating public disclosure rules. I and many others who have a firm understanding of how these systems work—by predicting next words from previous productions crossed with the question token stream—are quick to dismiss the claims of sentience. But what does sentience really amount to and how can we determine if a machine becomes sentient?

Note that there are those who differentiate sentience (able to have feelings), from sapience (able to have thoughts), and consciousness (some private, subjective phenomenal sense of self). I am willing to blend them together a bit since the topic here isn’t narrowly trying to address the ethics of animal treatment, for example, where the distinction can be useful.

First we have the “imitation game” Turing test-style approach to the question of how we might ever determine if a machine becomes sentient. If a remote machine can fool a human into believing it is a person, it must be as intelligent as a person and therefore sentient like we presume of people. But this is a limited goal line. If the interaction is only over a limited domain like solving your cable internet installation problems, we don’t think of that as a sentient machine. Even against a larger domain of open-ended question and answering, if the human doesn’t hit upon a revealing kind of error that a machine might make that a human would not, we remain unconvinced that the target is sentient.… Read the rest

Notes on Pumps: Sensibilities and Framing with Algorithmic Feedback

“A sensibility is one of the hardest things to talk about.” So begins Sontag’s Notes on “Camp” in the 1964 Partisan Review. And what of the political anger and disillusionment across the United States and in the developed world? What of the gnawing desire towards superiority and control that accompanies authoritarian urges? What of the fear of loss of power to minority ethnic and religious groups? These may be the most discussed sociopolitical aspects of our modern political sensibility since Trump’s election in 2016 when a bitter, vindictive, hostile, crude, fat thug briefly took the reigns of America, then pushed and conspired to oppose the election of his successor.

What attracted his followers to him? I never encountered a George W. Bush fanatic during his presidency. Though not physically small, he talked about “compassionate conservatism” with a voice that hung in the upper register of middle pitches for men. He was neither sonorous nor mean. His eyebrows often had a look of surprise and self-doubt that was hinted at in claims he was a very reluctant candidate for president. I met people who voted for him but they seemed to accept him as an acceptable alternative to Gore or, later, to Kerry—not as a figure of passionate intrigue. Bush Jr. did receive a rally-around-the-flag effect that was based on circumstances that would later bring rebuke over the casus belli of the Iraq War. Similar sensibilities were true of the Obama years—there was a low positivity for him on the Left combined with a mildly deranged antagonism towards him on the Right.

Was the lack of Trump-like animating fanaticism due to the feeling that Bush Jr. was a compromise made to the electorate while Trump was, finally, a man who expressed the real hostility of those who vote Republican?… Read the rest

We Are Weak Chaos

Recent work in deep learning networks has been largely driven by the capacity of modern computing systems to compute gradient descent over very large networks. We use gaming cards with GPUs that are great for parallel processing to perform the matrix multiplications and summations that are the primitive operations central to artificial neural network formalisms. Conceptually, another primary advance is the pre-training of networks as autocorrelators that helps with smoothing out later “fine tuning” training programs over other data. There are some additional contributions that are notable in impact and that reintroduce the rather old idea of recurrent neural networks, networks with outputs attached back to inputs that create resonant kinds of running states within the network. The original motivation of such architectures was to emulate the vast interconnectivity of real neural systems and to capture a more temporal appreciation of data where past states affect ongoing processing, rather than a pure feed-through architecture. Neural networks are already nonlinear systems, so adding recurrence just ups the complexity of trying to figure out how to train them. Treating them as black boxes and using evolutionary algorithms was fashionable for me in the 90s, though the computing capabilities just weren’t up for anything other than small systems, as I found out when chastised for overusing a Cray at Los Alamos.

But does any of this have anything to do with real brain systems? Perhaps. Here’s Toker, et. al. “Consciousness is supported by near-critical slow cortical electrodynamics,” in Proceedings of the National Academy of Sciences (with the unenviable acronym PNAS). The researchers and clinicians studied the electrical activity of macaque and human brains in a wide variety of states: epileptics undergoing seizures, macaque monkeys sleeping, people on LSD, those under the effects of anesthesia, and people with disorders of consciousness.… Read the rest

Triangulation Machinery, Poetry, and Politics

I was reading Muriel Rukeyser‘s poetry and marveling at some of the lucid yet novel constructions she employs. I was trying to avoid the grueling work of comparing and contrasting Biden’s speech on the anniversary of January 6th, 2021 with the responses from various Republican defenders of Trump. Both pulled into focus the effect of semantic and pragmatic framing as part of the poetic and political processes, respectively. Sorry, Muriel, I just compared your work to the slow boil of democracy.

Reaching in interlaced gods, animals, and men.
There is no background. The figures hold their peace
In a web of movement. There is no frustration,
Every gesture is taken, everything yields connections.

There is a theory about how language works that I’ve discussed here before. In this theory, from Donald Davidson primarily, the meaning of words and phrases are tied directly to a shared interrogation of what each person is trying to convey. Imagine a child observing a dog and a parent says “dog” and is fairly consistent with that usage across several different breeds that are presented to the child. The child may overuse the word, calling a cat a dog at some point, at which point the parent corrects the child with “cat” and the child proceeds along through this interrogatory process, triangulating in on the meaning of dog versus cat. Triangulation is Davidson’s term, reflecting three parties: two people discussing a thing or idea. In the case of human children, we also know that there are some innate preferences the child will apply during the triangulation process, like preferring “whole object” semantics to atomized ones, and assuming different words mean different things even when applied to the same object: so “canine” and “dog” must refer to the same object in slightly different ways since they are differing words, and indeed they do: dog IS-A canine but not vice-versa.… Read the rest

A Learning Smorgasbord

Compliments of a discovery by Futurism, the paper The Autodidactic Universe by a smorgasbord of contemporary science and technology thinkers caught my attention for several reasons. First was Jaron Lanier as a co-author. I knew Jaron’s dad, Ellery, when I was a researcher at NMSU’s now defunct Computing Research Laboratory. Ellery had returned to school to get his psychology PhD during retirement. In an odd coincidence, my brother had also rented a trailer next to the geodesic dome Jaron helped design and Ellery lived after my brother became emancipated in his teens. Ellery may have been his landlord, but I am not certain of that.

The paper is an odd piece of kit that I read over two days in fits and spurts with intervening power lifting interludes (I recently maxed out my Bowflex and am considering next steps!). It initially has the feel of physicists trying to reach into machine learning as if the domain specialists clearly missed something that the hardcore physical scientists have known all along. But that concern dissipated fairly quickly and the paper settled into showing isomorphisms between various physical theories and the state evolution of neural networks. OK, no big deal. Perhaps they were taken by the realization that the mathematics of tensors was a useful way to describe network matrices and gradient descent learning. They then riffed on that and looked at the broader similarities between the temporal evolution of learning and quantum field theory, approaches to quantum gravity, and cosmological ideas.

The paper, being a smorgasbord, then investigates the time evolution of graphs using a lens of graph theory. The core realization, as I gleaned it, is that there are more complex graphs (visually as well as based on the diversity of connectivity within the graph) and pointlessly uniform or empty ones.… Read the rest