Game Designed To Induce Synesthesia

Synesthesia

Jamie Ward , Julia Simner , in Multisensory Perception, 2020

Abstract

People with synesthesia have a remarkable way of experiencing the world in which numbers may evoke colors, words may trigger tastes, or the silent movement of an object may elicit a sound. In this chapter, we present an overview of the different kinds of synesthesia and the challenges faced in proposing a common explanation (and definition) of such a diverse collection of experiences. We focus specifically on developmental synesthesia, summarizing evidence that the brains of synesthetes have certain functional and structural differences, and discuss theoretical accounts as to how synesthesia might emerge developmentally. These accounts include both a role of a genetic component and environmental influences, where the latter seem to be particularly important for shaping the particular associations. Finally, synesthesia is linked to a distinct pattern of cognitive abilities affecting imagery, memory, and perception.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780128124925000139

How synesthesia may lead to enhanced memory

Radhika S. Gosavi , Edward M. Hubbard , in Multisensory Perception, 2020

Introduction

Synesthesia is a benign perceptual/cognitive variant in which stimulation in one sensory or cognitive stream leads to associated experiences in a second, unstimulated stream ^1,2 (see Chapter 13). For example, in one of the most common forms of synesthesia, grapheme-color synesthesia, letters or numbers are perceived as if viewed through a colored overlay. ^3,4 In another common form, ordinal linguistic personification, numbers, days of the week, and months of the year evoke personalities. ^5,6 In sequence-space synesthesia, numbers, months of the year, days of the week, and other ordered sequences are experienced as corresponding with precise locations in space, such as a three-dimensional view of a year as a map. ^7–9 "Colored hearing," which includes auditory word-color and music-color synesthesia, ^10–12 is one of the few types that involves prototypical cross-modal linkages. Although this form of synesthesia is often presented as a paradigmatic example of synesthesia, it is less common than many of the previously mentioned forms that involve cross-stream, but within-modality, associations.

Although it is often referred to as a "neurological condition," synesthesia is not listed in either the DSM-IV or the ICD classification, ¹³ as it generally does not interfere with normal daily functioning. Indeed, most synesthetes report that their experiences are neutral or even pleasant. ^14,15 Rather, we tend to think of synesthesia as being similar to color blindness or perfect pitch, a difference in perceptual experience. We sometimes refer to it as a neurological condition to reflect the brain basis of this perceptual difference, but this should not be taken to indicate that synesthesia typically yields functional impairments. The unusual reports of synesthetes may lead clinicians to think of synesthesia as a symptom of a psychiatric disorder, and indeed, the DSM-V revisions included discussions of including synesthesia as part of a differential diagnosis for other psychiatric conditions. ¹⁶ Its high prevalence rate means that congenital synesthesia sometimes may be found in patients who present with psychiatric conditions. Finally, despite this general pattern, emerging evidence suggests that synesthesia may be associated with a heightened rate of radiological markers for multiple sclerosis ¹⁷ and anxiety disorders. ¹⁸

Since the rediscovery of synesthesia in the mid-1980s, ^19,20 much of synesthesia research has focused on demonstrating the reality of synesthetes' experiences ^21–24 and exploring the neural substrates that lead to synesthetic experiences. ^25,26 Fortunately, as these foundational issues have become more settled, ¹ synesthesia research has expanded to address a much wider variety of topics. Synesthesia research has explored the impact of other life experiences on synesthesia, ²⁷ as well as the impact of synesthesia on perceptual and cognitive functions ^3,23,24 and on possible comorbidities between synesthesia and other cognitive and neural traits. ^17,18,28 In this chapter, we focus on the impact of synesthesia on memory, reviewing over 50 years of research on this topic, and building on our own recent investigations of how synesthesia impacts multiple memory processing stages.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780128124925000140

How do crossmodal correspondences and multisensory processes relate to synesthesia?

David Brang , Vilayanur S. Ramachandran , in Multisensory Perception, 2020

Abstract

Synesthesia is a perceptual phenomenon in which stimulation of one sensory modality evokes additional (usually) sensory experiences in an unrelated modality (e.g., sounds evoking colors). Much has been written about the definition of synesthesia, but as it remains a behaviorally defined phenomenon, definitional requirements are premature at this early stage (indeed, precision of word use often follows, rather than precedes, conceptual understanding). Synesthesia is thought to arise from either increased connectivity or reduced inhibition between associated sensory areas that usually do not interact. While the condition is typically studied in individuals who experience developmental variants of the condition, nonsynesthetes can experience analogous sensations via hallucinogens or as a result of sensory deprivation, ¹ raising the possibility that synesthesia exists as a latent feature in all individuals. Research has long sought to identify the relationship between synesthesia and more common multisensory interactions, including crossmodal correspondences (commonly agreed upon multisensory associations, such as small, white objects pairing with high-pitched sounds). In this chapter we review evidence both in support of and in opposition to models relating synesthesia and crossmodal correspondences, and suggest future research to distinguish between them. As a first approximation, we note that idiopathic synesthesia often involves apparently arbitrary correspondences (e.g., the color blue and the number 5 have nothing in common with one another) whereas multisensory associations usually "make sense" (e.g., an ameboid shape and a jagged shape resemble their auditory counterparts, the pseudowords bouba and kiki, respectively). Yet as we shall see, this distinction is not always true.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780128124925000127

Bouba-Kiki

Vilayanur S. Ramachandran , ... Chaipat Chunharas , in Multisensory Perception, 2020

Physiology of synesthesia

What causes synesthesia? To account for the phenomenon of synesthesia (grapheme color), we proposed the sensory cross-activation hypothesis ^6,7 which simply postulates the following: when neurons mediating one sense modality (e.g., form) are activated by a real stimulus, there is spontaneous cross activation of neurons that would ordinarily be mediating another modality (e.g., color) or attribute. Cross activation can occur due to abnormal anatomical connections or disinhibition of preexisting connections—including feedback projections—between brain modules. ²⁴

Why would such connections for cross activation exist? We postulated that in a fetus or early infancy, there are redundant connections between brain modules, perhaps even far-flung ones—which are then subsequently pruned under the control of specific, "pruning genes"—resulting in the characteristic modular organization in the adult brain. One example of this is the segregation of the grapheme and color areas of the fusiform gyrus. If there is a mutation that causes defective pruning, and the mutation happens to be expressed (due to transcription factors) in a specific brain location, for example, only in the fusiform gyrus, the result is grapheme-color synesthesia. The same model can account partly for more rare forms of synesthesia such as the man who tasted shapes, ²⁵ or "texture–emotion synesthesia," ²⁶ both of which could involve cross activation between the insula (taste/emotion) and secondary somatosensory cortex (S2; texture)—or the intraparietal sulcus (IPS) and the anterior parts of the supramarginal gyrus (SMG) (shapes).

As noted earlier, brain-imaging studies (fMRI), which allow precise localization of V4 activity, support our contention that "projector" synesthetes have cross activation between color and form in the fusiform gyrus. ¹⁸ Even more compelling evidence for our hypothesis is the recent finding—using DTI—showing that there is an actual increase of axons in the fusiform gyri of projector synesthetes, ^21,27 whereas, in associators, the increase is seen in the inferior parietal cortex, precisely as predicted by our model. ^6–8 Indeed, there is an overall increase in long-range resting-state potentials throughout the brains of synesthetes, ²⁶ and consistent with this, we have found that grapheme-color synesthetes show enhanced crossmodal interactions between modalities other than color and form (e.g., auditory and visual modalities ²⁸ ; see also Lacey et al., who showed enhanced Bouba-Kiki effects in synesthetes ²⁹ ). Thus, unlike neurotypical individuals, synesthetes as a whole appear to have a widespread enhancement of sensory—and perhaps conceptual—cross activation. ^6,7 In our scheme, this would also explain the increase, not only in creativity and memory but also in the persistent tendency of synesthetes to form strange associations between seemingly unrelated attributes and ideas. For example, "4 is a warm, even-tempered, puppy-dog kind of guy. A Labrador, if you will. He likes being outdoors and gives really good hugs. He recycles and rides bikes, but isn't all up in your face about it." This is a far cry from the anatomical precision that we postulated for certain types of synesthesias (e.g., grapheme color or number lines—see last section of this chapter). Such strange introspections as quoted above that lie at the boundary between sensation and cognition add a dimension of complexity to the synesthesia problem, which makes it at once both frustratingly challenging, yet providing a novel foothold to approaching some of the most enigmatic and treasured aspects of our minds. As Holmes told Watson, "It is this, that lifts this case from the regions of the commonplace."

Indeed, listening to some of the more outlandish remarks of synesthetes, one realizes that even though synesthesia has already begun to provide us with valuable insights into the mechanisms underlying brain function, we have barely scratched the surface of the problem.

Fulfilling at least a subset of the nine criteria (loosely defined) described earlier and the associated (presumed) activation of V4 appears to be true mainly for projectors (constituting 10%–15% of synesthetes). In associators, the color is evoked not by graphemes (visual shapes of numbers), but by their MEANING—such as the abstract idea of numerical sequence. This would encompass the ordinary sequence of natural integers as well as days of the week or months of the year (which many associators experience as being colored). We suggested that perhaps in many of these associators, the cross activation occurs higher up in the sensory processing stream. It is also worth noting that in some of them the color evoked by the first letter of a word tints the entire word (see below ^6–8 ).

But where is "higher up"? A reasonable candidate is the IPS—a cortical area that receives input from V4 ³⁰ and is adjacent to the angular gyrus (AG) (Fig. 1.5). The AG, as we shall see, represents numbers, fingers (as well as sequences they embody), and—possibly, in conjunction with the hippocampus—calendars (see last section). It also receives input from the area processing "form" (including graphemes) in the fusiform gyrus. Thus, there is plenty of opportunity for direct cross activation.

A second possibility is that unidentified back projections from the AG to V4 activate color-sensitive neurons. This indirect activation would explain the "weak" qualia experienced by associators in contrast to projectors ("I see the colors in my mind's eye" vs. "I see it painted on the letter"). Finally, in addition to the visual grapheme area in the anterior fusiform, there appears to be a visual word form area (VWFA) in the middle fusiform that encodes whole words including their pronunciation, and it's possible some associators have cross activation arising there.

The cascade of activation in different types of synesthesia can be further teased apart by MEG (which has exquisite temporal resolution). Experiments spearheaded in our laboratory by David Brang showed that V4 is activated 5 ms after the graphee area, rather than the much longer delay of the kind that would be required if back projections from the AG were involved. ¹⁹

We must be cautious, however, not to overemphasize the hardwired genetic approach to synesthesia. We are sometimes misquoted as having shown that "synesthesia is genetic"—a sentence that is biologically meaningless. None of us is born with graphemes. We have always maintained that what the synesthesia gene(s) confers is a propensity to link arbitrary shapes and colors.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780128124925000012

Ecstatic and Synaesthetic Experiences during Musical Perception

MACDONALD CRITCHLEY , in Music and the Brain, 1977

Timbre: harmonic composition: overtone-structure

Secondary sensations may also depend upon the mechanical origin of the musical sounds. Specific differences arise according to whether the notes emanate from a single instrument, a small chamber music ensemble, or from a full orchestra. According to Donath (1923), sounds proceeding from a piano, 'cello, harp, violin, flute and oboe evoked photisms which were violet, orange, wine-red, chrome yellow, blue and olive-green respectively. However, in Myers' subject, the colour-linkages were quite different. Thus, a 'cello, bassoon, horn, trombone, violin, and fife were related in a specific manner to brownish pink, brownish yellow, brownish rose, red, pink merging into blue, pale blue, or green.

In some other subjects it is not the individual notes, nor their overtonal structure, nor yet the factor of volume which produce the most explicit synaesthesia, but rather music in its role as a complete and coordinated entity. A personally observed patient with a right frontal meningioma, associated the sound of any military band with an image of "nasty red", and dance-music "of the nice quiet sort" with a misty blue. Scriabin, listening to a theme in D major, became aware of a photism of yellow, while to Rimsky-Korsakov, seated beside him, the same sound evoked a tint which was more golden. According to Myers (1914) Chopin inspired "translucent colours like green leaves in Spring" while the works of Schumann conjured up an opaque purple. Schubert's Third Impromptu was wine-red, while Beethoven's Laube Sonata Opus 27 No. 2 was blue (Donath). Of a more contrived character was the linkage of one of Schubert's Arias with "so sunny a warmth and so delicate a green" that Myers seemed to become aware of the scent of fir trees. Gounod (quoted by Fraser-Harris (1905)) went further, by implicating the language employed by the singer. To him French was not so colourful as Italian, though finer in hue. Pedrono and Ughetti observed a polyglot synaesthetic whose colour-associations varied according to the particular language employed.

In anecdotal vein it can be said that when Liszt was appointed to Weimar as Kapellmeister, he bewildered his players at rehearsals by urging … "more pink here, if you please"; or by declaring "that is too black"; or, "here I want it all azure".

The same melodic phrase may possess secondary sensations which change according to the key selected by the instrumentalist. That major and minor keys should differ in this context is not surprising. Some have spoken of minor keys being associated with grey or black photisms, major ones being green, blue, pink, or red. To some subjects, major chords are "brighter" than minor. But to those gifted with a more elaborate faculty of synaesthesia, highly specific differences may result according to the particular key chosen, whether it be major or minor.

As already stated, the simple frequency of the sound-waves may be an important and decisive determinant in colour-associations. In some persons the matter appears to go deeper. For example, middle C heard in acoustic isolation might bring about a certain colour-association; but were this same note to be sounded immediately after another note, say F sharp, then the colour-association would be different. Thus a note of 500 Hz following 1200 has been known to evoke a rosy brown, while after 600 the brown was much softer; the photism of 500 Hz alone was blue.

The deeper one delves into this problem, the richer the treasures that are salvaged. For example, a phenomenon of perceptual perseveration may be detected in some cases, or alternatively, an imagery which is astonishing in its accuracy. A musician might strike a single note on a piano, and then, back at home, he would later be able to tune his violin with absolute precision by the simple device of matching the colour-photism (Colman).

In aphasic subjects something similar may occur. A life-long synaesthetic became afflicted after a stroke with a marked anomia, and would be at a loss to put a name to an article presented to him. In his efforts he might utter various sounds which were phonemically analogous to the elusive term, but not quite accurate. That the utterance was wrong was obvious to the aphasiac, because to him its secondary colour-association was also wrong. In his search for the mot juste he was greatly assisted by his more rapid power of discovering the appropriate photism.

A case has been described of a couple of synaesthetics, both of whom associated a note with the pitch of G with the colour red. Neither subject, however, could identify this note when it was sounded alone. Both these individuals also happened to be afflicted with a red-green colour-blindness. Albertoni (1889), referring to these two persons, spoke of an "auditory Daltonism".

Then there is the unusual phenomenon which may be referred to as "sporadic synaesthesia". Bowers (1969) is an instance in point when he wrote "on two occasions I have seen radiant flashes of blinding colours and lights during performances of Scriabin's music. I was neither prepared for them, nor was I able to repeat them at any other time. They happened; I saw light unexpectedly and for no explicable or useful purpose. The experiences lasted for not more than a few seconds and were gone. They were quite different from a thrill of sensation, tears of pleasure, or usual emotions associated with beautiful music. I was more surprised than pleased. They have not recurred. But I have not forgotten them."

Chromatic sensations secondary to music seem to be independent of the deliberate conjuring up of colour-imagery. Thus, while a synaesthetic subject is directed to concentrate on a mental concept of redness, and then a note is struck on a pianoforte carrying with it some other type of colour-hearing, the two colours do not blend, but remain distant and apart. Schultze, on the other hand (1912) described a phenomenon of "reciprocal conditioning", whereby the subject, when surrounded by an environment of red, would modify his piano-playing almost insensibly so as to produce that type of music which was evocative of a secondary sensation of identical tint. Should, however, he feel himself to be in a mood of "blackness" he could deliberately "play himself out of that particular feeling-tone" into one of, say, redness.

At least one instance has been reported, again by Schultze, of tertiary sensations coupled with music. A man of 30 years had been aware of his synaesthesia since the age of ten, but his personal colour-hearing was also closely linked with gustatory associations. Thus, instrumental music would produce an imagery first of taste and then of colour, as if the stimulus "went from the ear through the mouth to the eye". Minor chords would in his experience give origin to a hard and bitter taste, and major chords one which was sweet and pleasurable. The distinction was so clear-cut that the subject could accurately identify the keys concerned. After listening for a time, he would feel "full up". He used to refer to "a mouthful of music" ("Mund voll Musik"). Perseveration would also occur, and when the music ceased, after-associations of taste and colour would linger on. The subject spoke of "digesting" the music ("die Musik verdaut"). One of the subjects described by Myers (1911) proclaimed that of a particular musical note … "I call it lilac … it rather suggests the taste of thick soup." In this same connection one is reminded, too, of Proust who spoke of clear sounds evoking blazing colours … something comparable to the perfumed silkiness of a geranium ("La soierie embaumée d'un géranium").

That decadent eccentric J.-K. Huysmans in his "À Rebours" afforded us a long description of the search for unusual sensory experiences by the voluptuary Des Esseintes. An elaborate assemblage of liqueurs constituted his orgue à bouche. Tasting now here, now there, he played "internal symphonies" which produced on his palate a series of sensations analogous to those with which music gratifies the ear. Each liqueur corresponded in taste with the sound of a particular instrument. Curaçao, for instance, was like the clarinet with its shrill, velvety note; kümmel like the oboe, whose timbre is sonorous and nasal; crème de menthe and anisette, the flute, at one and the same time sweet and poignant, whining and soft. Kirsch blew a wild trumpet blast; gin and whisky deafening the palate with their harsh blare of cornets and trombones; marc de bourgogne matched the tuba with its deafening din; whilst "roulent les coups de tonnerre de la cymbale et de la caisse frappés à tour de bras, dans la peau de la bouche, par les rakis de Chio et les mastics!"

Des Esseintes went further: the music of liqueurs had its specific scheme of interrelated tones. On his tongue he became able to perform voiceless melodies and mute funeral marches solemn and stately. At times he would compose melodies of his own, executing pastorals with the gentle blackcurrant ratafia that set his throat resounding with the mellow notes of warbling nightingales.…

The phenomenon of colour-hearing can be provoked or facilitated with particular liveliness in two very different circumstances. First, there are the states of intoxication from such hallucinatory drugs as marihuana, mescal and lysergic acid. References to pharmaco-phantastica go back many years. Thus, Gautier (1843), after experimenting with hashish, wrote "J'entendais le bruit des couleurs. Des sons verts, rouges, bleus, jaunes, m'arrivaient par ondes parfaitement distinctes."

Lewin (1924), an early writer upon such drugs, asserted that he could see, as well as hear, musical stimuli … "From the far distance came tones, spherical sounds."

The literature dealing with acute mescal intoxication is considerable. According to Mayer-Gross and Stein (1926), the beating of a metronome kindled in one of their subjects a synchronous play of form and colours. The note C conjured up a red, and F an orange-yellow, higher tones bringing about tints which were lighter and less defined. The beating of chimes induced an hallucination of purple. When the window-frame was struck by an iron bar, the green of the trees outside seemed to become rhythmically lighter in strict unison with each impact. The mescal-intoxicated subject was astonished. In his own words "Da haben wir die Magie". K. Behringer described in his monograph Der Meskalinrausch (1927) examples of Mitempfindungen evoked by this drug. "Unter Musikeinwirkung nahm diese Erscheinung mehr Bandform an, worauf die höhen Töne lebhafte grelle Farben und die tiefen Töne mehr stumpfe, matte Farben und mehr eintönige hervorriefen …".

The other condition which promotes synaesthesia is blindness. Many have quoted – though not always accurately – Locke's account of a congenitally blind man who, asked to explain what was his conception of the colour "red", likened it to the note of a trumpet. ^* It is well known that individuals who have been sightless for many years often become afflicted by obtrusive visual hallucinations specific in colour and constant in shape, though often strongly influenced by auditory stimuli. This topic was exhaustively dealt with by Wilhelm Voss of Kiel in his monograph Das Farbenhören bei Erblindeten (1930). The association of musical talent with blindness must surely be more than fortuitous, and synaesthesia may well constitute an important link, suggesting a type of compensation. The late Thomas Wood (1936), a blind musician of my acquaintance, once described to me his own life-long faculty of colour-association. "It brings a definite colour to single notes, to notes in groups, to movements; it changes the colour according to height or depth, scoring, key; over all this it lays a colour that goes with the work as a whole, and at times a shape is added which is just as fortuitous as the colours themselves." Different patterns and colour-qualities of these phenomena could be evoked by playing Mendelssohn's Spring Song in this key or that. In his autobiography the author stressed that colour-association is something which transcends mere metaphor or linguistic juggling.

This last point is important, for it is easy for one devoid of the gift of audition colorée to dismiss the whole matter as a mere literary contrivance stemming from a vocabulary which is either superabundant but poorly controlled, or on the other hand one which suffers from sheer verbal inadequacy. The metaphorical use of chromatic terms to describe auditory experiences is a literary commonplace, sometimes grossly overplayed. Oscar Wilde, for example, wrote that "her voice was exquisite … but it was wrong in colour". Elsewhere he referred to the vermilion lips of Salome … "like the scarlet blast of trumpets". Proust spoke of the "red and mysterious" appeal of the Vinteuil septet, and in another place he mentioned the "geranium scent" of the music. Still more strained is the following passage from Myro Brinig in The Copper City. "He was a brass band of a man and his voice packed with all the colours and sounds and thrills of all the instruments in Sousa's band. He spoke, and his voice soared, dived, flew, swam, crashed, and thundered. His voice was a brush that painted a drab canvas into a conflagration of colours galore, but his voice could be soft too and sweet and win women and soothe babies."

Much may be forgiven those masterly writers whose creative upsurges and whose quest for revealing associations – even if far-fetched – resort to such stylistic tricks as strained but arresting metaphors, evocations of imagery or memory which are barely tangible, and illusions ranging from a déjà vu through a presque vu to a thrilling pas encore vu. But with many musical critics the problem is different. Their writings are not necessarily endowed with any conspicuous literary merit. Perforce they are compelled to draw upon the language of one art-form to describe the qualities of another. To the sensitive reader who happens to be musically deprived, such licence may become distasteful. All too often a veritable jargon results, as irritating as any other trade-language. Just as art critics borrow such musical terms as "tone", "scale", "loud", "soft", so journalists of music have not hesitated to refer to "brilliance", "form-hues", "tone-colour", and, of course, the fundamental term "chromatic".

Three other matters arising out of colour-hearing associations may be briefly touched upon.

The first refers to the occasional attempts by professional composers to interpolate within their orchestral works some other special sense modalities – most commonly visual, but olfactory too. That eccentric innovator, Scriabin – musician, mystic and theosophist – introduced within his Prometheus, the Poem of Fire, a special colour-scale and notation, which he described as a Clavier à lumières or Tastiera per luce. This work, conducted first by Koussevitsky, later by Altschuler, has been rendered but rarely and with very mixed reactions except in the U.S.S.R. The critics in America were unimpressed and disapproving. Undismayed, yet not content, Scriabin had the further ambition to compose a work which combined musical sounds not only with a play of light, but also with a medley of olfactory titillation. This was the basis of his elaborate work Prefatory Action and Mysterium, which has never yet been performed, for its completion was prevented by the composer's death. As a matter of fact, Scriabin had been anticipated in 1891, when a piece called The Song of Solomon was played in Paris giving "simultaneous appeal to the eyes, ears, and nose" (Philip Hales). The book was by Paul Rionard, the musical adaptations by Flamen de Labrely. The following year, New York witnessed an "experimental perfume concert" entitled A Trip to Japan in Sixteen Movements.

Arising naturally from the foregoing is the second consideration. I refer to that deliberate contrivance whereby sound and light are artificially combined in such a way as to form a novel art-form. This topic is not without intrinsic interest, and has been admirably documented by Adrian Klein in his classic Colour Music, the Art of Light (1926). The story

begins with Arcimboldo's "calorific music" in 1591, followed in 1720 by the invention of the Jesuit Louis Castel of a clavessin oculaire. Next in time came the works of D. Jameson, 1844, and H. R. Haweis, 1875. In 1893, A. W. Rimington constructed a colour-organ. In 1900 appeared Louis Favre's monograph La musique des couleurs et des musique de l'avenir. Then came the clavilux designed by Thomas Wilfrid in 1916. Nor must one overlook the contributions of Mrs. Greenewalt (1918) in New York, A. B. Hector (1922) in Sydney, Mr. Luckiesh of the American General Electric Company, and more recently the painter Kandinsky.

In the opinion of Ernest Newman, the claims – or perhaps it should be the hopes – that colour and music can be "translated" into each other, or that an art of "mobile colour" can be created as the visual equivalent of an art of mobile sound, rest on nothing but a pathetic fallacy.

However quaint or intriguing, this topic of artificial colour-music signifies but little within the province of the psychology of music and still less its neurology.

In the context of synaesthesia, it is not enough merely to describe, for it is necessary to venture beyond mere reportage and to ponder over the possible origins of this phenomenon.

Two main hypotheses exist. The first, coupled with the name of Peillaube (1904), is relatively simple. It subsumes that colour-synaesthesia is the product of a chain of mental associations, some of the intermediate links having dropped out of awareness. Thus the familiar story of trumpet-blasts provoking a photism of red, may stem from the fact that such a sound immediately culls up in some persons an imagery of soldiers on parade. Ordinarily they will be in dress-uniform. This evokes a mental picture of scarlet. Should the middle part of this notion eventually become submerged, there will remain a synaesthetic linkage of trumpet-calls with redness.

An alternative theory regards colour-hearing – and for that matter colour-taste – as a perpetuation of some primitive perceptual experience, a sort of amorphous Urempfindung, which later in evolution becomes elaborated, differentiated and dissociated along the channels of the various special senses (Myers; Wundt). This idea implies a basic unity of these special senses, and when it comes to communication, a poverty either of cognition, or of vocabulary, or of both. Thus, in the French language, the verb "sentir" means not only to feel but also to smell and touch, while in German, the adjective "hell" (bright) was originally an auditory attribute, but its usage has spread so as to apply to vision. At least one African tongue utilises a special word for "seeing", but in addition, only one other term to stand for "hearing", "touching", "smelling", and "tasting".

The intriguing doctrine of the unity of the senses probably took origin in an arresting albeit obscurely written work by E. M. von Hornbostel which first appeared in Melos, Zeitschrift für Musik, Berlin (1925). The author envisaged that a sensuous state exists which is not limited to one single sense – a super-sensuous sense-perception as he called it. The essential component of the "sensuous-perceptible" is not that which separates the senses from one another, but that which unites them; "unites them among themselves; unites them with the entire (even with the non-sensuous) experience in ourselves; and with all the external world that there is to be experienced".

The same idea was discussed by H. Kleint of Luxembourg (1940) and by E. G. Boring (1942), and by E. A. Hayek in 1952 in depth. Hans Werner also examined critically this conception in his volume on the Comparative Psychology of Mental Development (1948). Hayek's contribution was of particular value. He observed that these intersensory and intermodal relations are relatively inconspicuous, and that with the development of conceptual thought, and the weighty influence of sensualism in scientific thought, these phenomena tend to be suppressed. Contrary to the view of Max Planck that intermodal associations are the product of our reflective powers of imagination, Hayek believed that the opposite was the case. It is sophistication which makes us overlook that which is obvious to naïve experience. Synaesthesia is not a linguistic matter of metaphor, but is the outcome of genuine intersensory attributes, which may at times be so dynamic as to lead to phenomena such as colour-hearing. Hayek believed that the employment of transmodal metaphors in speech is something more than a turn of phrase, being the product of veritable perceptual attributes of an intersensory kind. These may occasionally be so strong that sensations belonging to one modality may regularly be accompanied by others which belong to some other modality, as in the case of colour-music. Synaesthesia of this sort is especially vivid during the earlier stages of mental development. Although hesitant to go all the way with Boring, Hayek asserted that all mental qualities – directly or indirectly – are so interrelated that any attempt to give an exhaustive description of any one of them would make it necessary to describe the relations existing between all of them.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780433067030500206

To Cognize is to Categorize

Stevan Harnad , in Handbook of Categorization in Cognitive Science (Second Edition), 2017

2.21 Recoding and Feature Selection

Is there any way to increase our capacity to make categorizations? One way is to add more dimensions of variation. Presumably this is one of the ways in which S's synesthesia helped him. But even higher dimensionality has its limits, and its power of resolution never approaches the granularity of the JND of sensory discrimination.

Another way of increasing memory is by recoding. Miller showed that if we have to remember a string of 0s and 1s, then remembering a string of about seven items is our limit. If, however, we first learn to recode the digits into, say, triplets in binary code, using their decimal category names—so that 001 is called one, 010 is called two, and 011 is called three, etc., and we overlearn that code, so that we can read the strings automatically in the new code, then we can remember three-times as many of the digits. The seven-item limit is still there, but it is now operating on the binary triplets into which we have recoded the digits: 101 is no longer three items: it is recoded into one "chunk," called five. We have learned to see the strings in terms of bigger chunks—and it is these new chunks that are now subject to the seven- (or four-) item limit, not the single binary digits (bits).

Recoding bits of information by overlearning bigger chunks is a way to enhance rote memory for sequences. Something similar operates at the level of features of objects: although the number of features our sensory systems can detect in an object is not infinite, it is large enough so that if we see two different objects that share only one or a few features, we will not necessarily be able to detect that they share features, and, hence, that they are the same kind (category of object). This is again a symptom of the "underdetermination" mentioned earlier. It is related to the so-called "credit-assignment problem" in machine learning: how to find the winning feature or rule among many possibilities (Fu & Anderson, 2008; Sutton, 1984)?

To be able to abstract the shared features, we need supervised categorization training (also called "reinforcement learning"): trial and error, guided by corrective feedback, based on a large enough sample to allow our brains to solve the credit-assignment problem and abstract the invariant features underlying the variation, the features that distinguish the members of the category from the nonmembers. The result, if the learning is successful, is that the inputs are recoded, just as they are in the digit-string memorization; the features (or dimensions) are reweighted. The invariant features of the members of the same category are given a heavier weight than all the remaining irrelevant features. The objects that are of the same kind, because they share invariant features, are consequently seen as more similar to one another; and objects of different kinds, not sharing the invariants, are seen as more different (Carvalho & Goldstone, 2016; Folstein, Palmeri, Van Gulick, & Gauthier, 2015).

This within-category enhancement of perceived similarity and between-category enhancement of perceived differences is again the CP described earlier in the case of color (Hanley, 2015). The sensory "shadows" of light frequency, intensity, and saturation were recoded and reweighted by our evolved color receptors, so as to selectively detect and enhance the spectral ranges that we consequently see as red, yellow, etc.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780081011072000026

Audiovisual crossmodal correspondences

Charles Spence , K. Sathian , in Multisensory Perception, 2020

Crossmodal correspondences and synesthesia

Over the years, there has been a long-standing tendency to want to link the crossmodal correspondences or at least the more surprising of them (such as the pitch-size correspondence), to synesthesia proper (see also Chapters 12 and 13 Chapter 12 Chapter 13 ). One way in which this occurs is when researchers refer to crossmodal correspondences as "synesthetic correspondences" (e.g., Refs.19,24,41,42,116) or as "synesthetic associations." ⁵⁹ Some researchers, though, have gone further in describing crossmodal correspondences as a weak form of synesthesia ^117,118 (see also Refs.31,119,120). ^j Others, meanwhile, have argued vociferously against any such conflation of the two phenomena (see Refs.14,89,121,122 for reviews), interesting though both undoubtedly are in their own right. ^k

It is worth stressing here the many important differences separating synesthesia from the crossmodal correspondences. So, for instance, while the synesthetic relation between inducer and concurrent tends to be fixed across the lifetime in synesthetes, many crossmodal correspondences update as a function of the changing statistics of the environment or rather of the environmental stimulation that we happen to be exposed to (e.g., Ref.124). That said, it is interesting to note how recent research suggests that synesthesia may actually follow something of a similar pattern. For instance, children in primary school who experience letter-color synesthesia demonstrated an increase in the number of their synesthetic associations over a year of follow-up. ¹²⁵ Intriguingly, the associations appear to be tuned according to the statistics of the environment (including such factors as the frequency of letter usage, semantic, and ordinal associations), resulting in biases for more similarly shaped and similar-sounding letters to have similar colors associated with them. ¹²⁶

Another important difference is that in synesthetes the relation between inducer and concurrent tends to be absolute, whereas many of the most frequently studied of typical crossmodal correspondences are relative in nature—that is, it is the higher-pitched of two sounds that will be matched with the smaller of two circles, for example. ⁸⁹ Furthermore, while synesthesia, by definition, gives rise to an idiosyncratic conscious concurrent (at least according to Grossenbacher and Lovelace's ¹²⁷ definition), the typical correspondences seem to be characterized by a feeling that operates without the need for conscious concurrents. That said, some commentators have recently started to question the need for synesthetic concurrents to be conscious (see Ref.128), and one might wonder whether those with particularly vivid crossmodal mental imagery might not also experience some kind of concurrent based on the correspondences. ^129,130 In this context, it is relevant to note that synesthesia has been associated with enhanced vividness of visual imagery ¹³¹ and greater use of mental imagery. ¹³²

Given these, and the many other differences between synesthesia and the correspondences, one might reasonably expect that the neural substrates associated with the two phenomena would also differ; however, the limited evidence gathered to date does not permit a clear answer to this question. For instance, a TMS study designed to interfere with crossmodal correspondences and one aimed at knocking out synesthetic concurrents were both applied to parietal cortex ^74,133 (see Ref.134 for a discussion). Specifically, Bien and her colleagues ⁷⁴ attempted to disrupt the function of cortex in the right intraparietal sulcus using TMS over the parietal site P4 (an electrode location in the standard clinically used EEG electrode montage) and temporarily eliminated the effect of the crossmodal correspondence between auditory pitch and visual size on the spatial ventriloquism effect. Esterman and colleagues ¹³³ targeted TMS to the right angular gyrus at the junction of the intraparietal and transverse occipital sulci, using a neuronavigation system that allowed precise targeting of this anatomical locus in each participant and found that the Stroop effect typically elicited by the synesthetically induced color was eliminated by TMS (see also Ref.135). Unfortunately, due to the differences in the methods, the sites in the two studies cannot be directly compared, precluding judgment on whether the same or different processes were involved in the two studies.

Those who wish to connect the phenomena of synesthesia with the crossmodal correspondences often point to the similarity between certain synesthetic concurrents and the correspondences that have been demonstrated in nonsynesthetes (e.g., see Refs.136,137). At this point, it is perhaps worth remembering that synesthetes presumably experience crossmodal correspondences just like nonsynesthetes. ^l Perhaps the more interesting question here, though, is whether they are more sensitive to the effect of certain crossmodal correspondences than are nonsynesthetes. One might, a priori, expect stronger crossmodal correspondences for synesthetes in the domain, or modalities, in which they experience synesthesia (i.e., between the modalities of the inducer and of the concurrent). Interestingly, though, stronger crossmodal correspondences, as indexed by the strength of the IAT congruency effect, were found in synesthetes compared with nonsynesthetes for sound-symbolic crossmodal correspondences (between rounded/pointed visual shapes and the auditory nonwords "lomo" and "keekay"), but not for the lower-level crossmodal correspondences tested, i.e., pitch-elevation and pitch-size. ⁴⁶ This suggests that indeed synesthesia may be associated with a greater tendency to form sound-symbolic correspondences, sound symbolism being a domain outside those of the types of synesthesia reported by the participants. It is worth noting that almost all the synesthetes in this study were associators (whose synesthetic experiences are reported to occur in the "mind's eye"); thus, it remains an open question if projectors (whose synesthetic experiences are reported to occur out in the world where the inducing stimuli are perceived) might show stronger low-level correspondences in addition to sound-symbolic ones. ¹⁴²

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780128124925000115

Drug Addiction

Harald Sontheimer , in Diseases of the Nervous System (Second Edition), 2021

4.6 Psychedelic Drugs (LSD, Mescaline, Psilocybin)

The word psychedelic translates to "mind-manifesting" or "soul-revealing." This family of drugs includes several nature-derived alkaloids; the most well-known are mescaline (peyote cacti), psilocybin (magic mushrooms), and LSD (ergot fungus). They typically cause hallucinations, dissociation from reality, physical euphoria, and feeling of weightlessness along with a wide-ranging effect on our senses and sensory integration. Changes in color perception, pattern recognition, auditory distortions, and even synesthesia (a crossover of a sense being perceived as another, such as hearing in colors) are reported. They also profoundly alter the senses of self and connection to other human beings. LSD was widely used among musicians and artists who describe that LSD increases their emotional response to their music or art. Note that psychedelic drugs are typically considered nonaddictive although tolerance may develop with continued use. An excellent account of the sensations triggered by LSD may be gleaned from the dairy of the plant chemist Dr. Albert Hofmann, who first isolated LSD in 1938 and came into contact with the drug during refinements of his procedures and later self-administration.

"Last Friday, April 16, 1943, I was forced to interrupt my work in the laboratory in the middle of the afternoon and proceed home, being affected by a remarkable restlessness, combined with a slight dizziness. At home, I lay down and sank into a not unpleasant intoxicated-like condition, characterized by an extremely stimulated imagination. In a dreamlike state, with eyes closed, I perceived an uninterrupted stream of fantastic pictures, extraordinary shapes with intense, kaleidoscopic play of colors. After some two hours this condition faded away." ²⁹

How psychedelic work is not well understood, in large parts because these drugs are Schedule 1, and like cannabis, they are challenging to use for research purposes. Generally speaking, psychedelic drugs mimic serotonin and are structurally similar to serotonin as illustrated for LSD in Figure 2. Classical psychedelics all have agonist activity at 5-HT2A receptors, which are G-protein-coupled receptors that generally increase neuronal excitation. 5-HT2A receptors are found pre- and post-synaptically in cortical pyramidal neurons that are innervated by serotonergic neurons originating in the dorsal raphe nucleus (DRN). These serotonergic neurons also project to the amygdala and hypothalamus. It is generally believed that 5-HT2A activation causes glutamate release from these cortical neurons and that this contributes to hallucinations. Indeed, blocking 5-HT2A inhibits hallucinations elicited by these psychedelic drugs. The targeted pyramidal cells in layer 5 are involved in sensory processing and show altered excitability upon 5-HT2A activation, which may explain sensory hallucinations such as hearing sounds and seeing vivid colors. Note that 5-HT2A receptor activation is also associated with increased release of oxytocin from the pituitary, which would explain the increased interest in bonding with other people when on psychedelic drugs.

Psychedelic drugs have many additional targets. Most bind with various affinity to 5-HT1, 3, 5, 6, and 7 and can prolong serotonin activity by inhibiting serotonin reuptake through the sodium-dependent serotonin transporter (SERT). In addition, some psychedelics, for example, psilocybin, alter dopamine signaling by acting as D2R agonist. These additional drug targets explain the complex biology exhibited by psychedelics, which cannot be explained by 5-HT2A activation alone.

Note that all serotonin receptors have a tendency to be removed from the membrane by endocytosis upon chronic stimulation. In fact, a single use of a psychedelic drugs causes a profound reduction of the membrane-associated receptors. This creates tolerance, as a drug user must use steadily increasing drug doses to achieve a comparable drug effect.

Clinical experimental use of psychedelic drugs had their heyday in the 1950s and 1960s where they were used in combination with psychotherapy to treat numerous neuropsychiatric illnesses. In the mid-1960s, recreational use of psychedelics, particularly LSD, became synonymous with the counterculture movement, resulting in a stigma that has persisted. LSD was made illegal in 1966, and all psychedelics were classified as Schedule 1 when the Controlled Substances Act was approved in 1970. This scheduling prohibited their medical use and resulted in few studies that explored their potential clinical benefit. However, recently interest in these compounds has resurfaced, and a few small clinical studies have shown promising effects in the treatment of obsessive–compulsive disorder and major depression, and particularly treatment of depression and anxiety associated with late-stage terminal cancer. In 2018, the FDA granted "breakthrough therapy" designation for psilocybin therapy for treatment-resistant depression. The resurgence in research allowed scientists to address common misconceptions that psychedelics increase risk of schizophrenia and suicide. This has been disproven by two large-scale population studies, which in fact found suicide rates to be decreased by over one-third in drug users. ³⁰

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780128212288000123

Ethical Dimensions of Commercial and DIY Neurotechnologies

Imre Bárd , in Developments in Neuroethics and Bioethics, 2020

3.2 Algorithmic coupling systems

This section will consider two broad types of technologies in the category of advanced devices, which rely on an algorithmic coupling system: sensory substitution and augmented reality.

3.2.1 Perceptual enhancement via sensory substitution

Sensory substitution devices translate information from one sensory modality into another (Renier & De Volder, 2013). In the 1960s, neuroscientist Paul Bach-y-Rita was investigating brain plasticity by studying whether it is possible for congenitally blind individuals to acquire the sense of vision. Together with colleagues, he conducted a series of ground-breaking experiments, which demonstrated the phenomenon of sensory substitution and established a new field of research. Bach-y-Rita's and colleagues' novel experimental setup consisted of a TV camera, a monitor, an electronic commutator device, and an array of 400 vibrating metal pins arranged in a 20 × 20 grid, which was attached to the back of a modified dentist's chair. The camera captured images of objects placed in front of it and the commutator device converted the visual signals into patterns of vibrations that a person sitting in the chair could feel in their lower back. The location of the vibrating pins was mapped unto the location of pixels on the monitor, and the strength of the vibration was proportional to the luminance of pixels on the screen, such that brighter pixels produced a stronger vibration (Bach-y-Rita, Collins, Saunders, White, & Scadden, 1969). This was the first sensory substitution device (SSD), which came to be known as the Tactile-Vision Substitution System (TVSS). After a few hours of training, users seated in the chair could recognize a range of objects, and judge their absolute size and distance based on the patterns of vibrations in their lower back (Bach-y-Rita et al., 1969). Importantly, subjects experienced the location of objects to be in front of them, despite the stimulus being presented to their backs (Guarniero, 1974). Bach-y-Rita also reported that upon zooming of the camera's lens subjects instinctively moved their head and body backwards, as if the object on the screen was moving toward them (Bach-y-Rita, 2002). Although there had already been a number of devices as early as the 1910s that relied on the concept of representing visual information as sound or touch, these were developed only in the context of allowing the blind to read and were considered reading machines (Koestler, 2004). In contrast, Bach-y-Rita's experiments opened up a much broader horizon by asking whether humans could learn new sensory systems.

Since then, a variety of sensory substitution devices have been developed. The Tongue Display Unit (TDU) converts visual information into patterns of small electrical stimulations delivered via a grid placed on the tongue (Sampaio, Maris, & Bach-y-Rita, 2001), capitalizing on its extremely high spatial acuity (Van Boven & Johnson, 1994). Another device, called the vOICe (capitalized letters representing "Oh, I see"), attempts to translate visual information into sound by using a head-mounted camera and a small computer that maps the visual field at regular intervals and creates a corresponding soundscape (Meijer, 1992). Long-term users of the vOICe learn to recognize the patterns in the soundscape and can develop great proficiency in navigating their environment using the device, even acquiring vision-like experiences (Ward & Meijer, 2010). The vOICe is available as a smartphone application as well. ¹ While existing vision-to-sound SSDs are only capable of mapping the spatial features of a visual scene, such as an object's shape, distance and movement, current research is building on insights from synesthesia in order to incorporate color information as well ( Hamilton-Fletcher, Mengucci, & Medeiros, 2016).

In 2015, Scott Novich and David Eagleman presented a sound-to-touch sensory substitution device called the Versatile Extra-Sensory Transducer (VEST) (Eagleman, 2015). The device is intended to assist individuals with impaired hearing and comes in the form of a vest, which can be worn underneath one's clothing. It translates sounds picked up via a microphone into patterns of tactile stimulation along the torso (Novich, 2015). With some training, deaf individuals can use the device to recognize spoken language. More recently, the company co-founded by Eagleman and Novich released the Neosensory Buzz, a wristband that also translates environmental sounds into vibrational patterns. According to the company's website, with practice the associations between sounds and vibrations become automatic and "a new sense is born." ²

So far, the chemical senses have not proved amenable to this kind of technology. For the sense of smell, it has been suggested that the reason might be the largely synthetic nature of olfaction. In other words, there do not appear to be primary building blocks or basic components to which olfactory experience could be reduced (Wilson & Stevenson, 2006). In the case of gustation, although we know that multisensory integration plays an important role in our perception of flavors (Spence, 2015), it is unclear how a mapping of the five basic tastes to sounds or tactile stimuli could produce an experience comparable to taste qualities in a person who never experienced flavors (Macpherson, 2018). Similarly, while there is great interest in haptic technologies at the moment (Jewitt, Price, Leder Mackley, Yiannoutsou, & Atkinson, 2020), my research has not uncovered any SSDs that would translate touch into a different sensory modality.

However, despite bold visions with regard to the potential of sensory substitution-based technologies, their uptake as assistive technologies to date has been extremely limited. Critics have argued that fundamental imbalances with regard to the cortical area available to tactile and auditory processing as compared to vision make it unlikely for these modalities to truly compensate for the lack of sight. Moreover, sensory substitution devices are incapable of capturing the hedonic aspects of the sense modality being substituted for, which might further explain their limited use (Spence, 2014, 2018).

Although SSDs emerged in the context of restorative and assistive technologies, since the early days of this research there has been interest in applying them for augmentation purposes (Bach-y-Rim, Webster, Tompkins, & Crabb, 1987). It has been proposed that the principle of sensory substitution may be extended to feed new kinds of information to the nervous system, thereby allowing for the acquisition of new senses. To illustrate the concept, a German research group developed a wearable device to deliver tactile stimulation around the waist to signal the direction of north, giving wearers constant information about their orientation in space. Through a series of experiments they demonstrated that "newly acquired sensory information can have profound effects on performance and perceptual experience" (Nagel, Carl, Kringe, Märtin, & König, 2005:R15). Similarly, the VEST developed by Novich and Eagleman has been proposed as a model for creating non-invasive devices to extend human perception and extract features from a variety of high-dimension data by leveraging neuroplasticity and the brain's remarkable pattern recognition ability (Novich, 2015).

3.2.2 Augmented reality

The concept of augmented reality (AR) seems like a suitable synonym for sensory augmentation or perceptual enhancement, nevertheless it denotes a certain set of technologies that are quite distinct from the sensory substitution-based methods discussed in the previous section. The term itself was coined in 1992 by Tom Caudell and David Mizell to describe the technology of overlaying computer-generated graphics on the real world (Caudell & Mizell, 1992). According to an early, operational definition by Milgram and Kishino, augmented reality refers to "any case in which an otherwise real environment is "augmented" by means of virtual (computer graphic) objects" (1994). Thus, along the reality-virtuality continuum—also proposed by Milgram and Kishino (1994)—augmented reality sits between the real physical environment and fully immersive virtual reality. Although this suggests that AR can only be visual, there are auditory and haptic AR applications as well (Romanus et al., 2020; Stone & Tata, 2017). Finally, Azuma (1997) offered three criteria to delineate AR, which state that AR combines the real and the virtual, it is real-time and interactive, and it is registered in 3D.

The most prominent form of augmented reality is visual, and it may be displayed in various ways, such as projected onto surfaces or viewed via screens, glasses, head-mounted displays, mobile phones, contact lenses or projected straight into the eye via virtual retinal displays or light field displays. Generating AR content involves Simultaneous Localization and Tracking (SLAM) methods as well as depth tracking technology. These are used to create a live digital representation of the user's environment, while also keeping track of the precise location and movement of the user in that mapped space in order to place virtual objects accurately. At present, we may distinguish between four main types of AR (Edwards-Stewart, Hoyt, & Reger, 2016; Thinkmobiles, n.d.):

-: marker-based AR, where an image, such as a QR code or a 3D-mapped object serves as the trigger for activating the AR content. For example, a picture printed on a breakfast cereal box, or indeed the whole box itself, when viewed via an AR device, might act as a trigger to activate the AR content;
-: markerless AR, which relies on GPS coordinates and the device's compass and gyroscope to position the digital content, such as a navigation support app that overlays arrows and other directional cues and information on a live view of the user's environment;
-: projection-based AR, where light is projected onto a surface to make it responsive to interaction, for example, augmenting a physical object such that touching certain parts can trigger digital events (i.e., turning an ordinary desk surface into a virtual keyboard);
-: superimposition-based AR, where a digital image is partly or completely superimposed on an object, for example, superimposing an internal view, such as an X-ray or other medical scan on a live view of a patient.

An important figure in the history of AR is Steve Mann, who has been described as the father of wearable computing. Mann holds several patents related to image processing, wearable computing, brain-computer interfaces (BCIs) and other technologies, and he is among the founders of the BCI company, InteraXon, and AR headset manufacturer, Meta. While Mann is an unusually prolific and influential inventor, his work illustrates how AR intersects with a number of other technologies, such as BCIs and the Internet of Things, and is part of a larger ecosystem of digital technologies. As a pioneer of the field, Mann has been wearing some version of computerized eyeware during most waking hours over the past 40 years (Mann, 2013). The Eyetap device allows Mann to enhance his vision giving him an ultra-high dynamic range view of the world (Mann et al., 2012) and allows him to filter out certain parts of his visual scene, which Mann has been using to remove ads for cigarettes in real-time (Bilton, 2012). Mann is critical of conventional uses of AR as digital pop-up messages thrown on top of one's field of view, which merely results in information overload. Instead, he prefers to use the term mediated, or aug-mediated reality to describe the ways in which his eyewear enables him to experience reality more fully (Mann, 2013). Mann's vision for human-computer interaction, which he has been practicing since the 1970s is rooted in the conviction that natural computer interfaces should twist around and adapt to the human, instead of forcing humans to bend over uncomfortable and clunky input methods like keyboards and mice (Computer History Museum, 2019).

Today, consumers mostly experience mobile AR, which uses the camera of a smartphone to overlay digital content on the live view of the physical environment on screen. AR-based applications are among the fastest growing segments of the smartphone app industry, with blockbusters like Niantic's Pokémon Go—an AR game in which users collect and interact with digital characters appearing in various real-world locations—grossing over $3 billion between 2016 and 2019 (Statt, 2020).

Mobile AR may be seen as an interim step before the arrival of everyday smart glasses, similar to those pioneered by Mann. These allow for the fluid and continuous blending of digital content with one's physical environment. Google's prototype smart glasses, released in 2013, ran up against considerable public resistance as people felt their privacy was being intruded upon by the device's onboard camera that could be used to overtly take photos and videos (Sarpu, 2014). However, the product was subsequently rebranded and is currently widely used in the context of specific industrial applications (Naughton, 2017). Similarly, Microsoft's HoloLens is primarily used for guided instruction programs in various industries from medical training to factory assembly, where rich, animated content is overlaid directly on a trainee's field of vision (Gavish et al., 2015). Currently available headsets are often bulky and not yet suitable for everyday use. However, the technology's development points toward continuously shrinking the device to a comfortable form factor. Over the past years, the size of augmented reality glasses and headsets has reduced significantly, while their computational capacity has continued to increase. Apple and Facebook are expected to release consumer smart glasses by 2023 and both companies hold a number of patents in this area of innovation, with significant recent hires and acquisitions (Robertson, 2019; Vincent, 2019). In early 2020 Mojo Vision showcased a prototype AR contact lens that seeks to bring the technology even closer to the body by realizing what the company calls "invisible computing." The product received the FDA's Breakthrough Device Designation and Mojo Vision intends to first release it as a medical device before rolling it out to consumers (Heater, 2020).

Augmented reality is expected to become drastically more realistic and seamless with the arrival of 5G connectivity and edge computing. This will allow for extremely fast data connections between network servers—which perform the calculations and render photorealistic visuals at high speed—and the smart glasses/contact lenses which display the content itself. Building on such advances, several companies, including Google and Facebook, are working on AR cloud technology, which will act as an underlying digital content layer mapped unto objects and physical locations. The AR cloud is regarded as the infrastructure for the dawn of a novel, spatial computing paradigm, which will open entirely new forms of interaction, new services and vast new monetization opportunities (Koetsier, 2019). Importantly, augmented reality technologies will be integrated with the Internet of Things and be powered by the rapidly advancing capabilities of AI and machine learning, resulting in a new ecosystem of technologies that enable our physical environment to be constantly enriched by an additional layer of digital information. Besides major technology companies working to advance this, there is also an open source effort driven by the Open AR Cloud Association, a US-based non-profit, ³ that seeks to ensure that this new infrastructure of experiencing reality is built on the values of freedom, transparency, privacy and dignity.

If and when large tech companies release AR glasses or contact lenses that are fit for everyday use, then this type of device can rapidly become the most widely used neurotechnology by a considerable margin. In 2019 alone, Apple sold over 185 million iPhones (Graham, 2020) and its sales of over 31 million Apple Watch units exceeded the performance of the entire Swiss watch industry (Reichert, 2020). Thus, the size of the market for these technologies, and hence the potential scope of their impact is enormous.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/S258929592030014X

Substance-related disorders

Jahangir Moini , ... Anthony LoGalbo , in Global Emergency of Mental Disorders, 2021

Hallucinogen use disorder

Hallucinogen use disorder involves the usage of drugs that include lysergic acid diethylamide (LSD), mescaline, and psilocybin. These drugs cause extremely unpredictable and idiosyncratic reactions, usually involving hallucinations, perceptional alterations, judgment impairment, depersonalization, and ideas of reference. Hallucinogens affect every individual in different ways. LSD is derived from a fungus that often contaminates rye and wheat flour. Mescaline is derived from the peyote cactus. Psilocybin is derived from several different types of mushrooms. Newer synthetic designer drugs have been produced, which are usually based on phenylethylamine or tryptamine molecules. Also, many illegal drugs contain other drugs of abuse, including ketamine, anesthetics, phencyclidine (PCP), and dextromethorphan. Some other drugs, such as marijuana, also have some hallucinogenic properties, though actual hallucinations may not occur. Many people also refer to hallucinogens as psychedelics.

Clinical manifestations

Hallucinogen intoxication causes alterations of perceptions. Users refer to these various effects as a "trip." Intense psychologic effects can alternate with periods of lucidity. The perceptional changes include the following manifestations:

•: Changes in mood—usually, these are euphoric, but can be depressive
•: Depersonalization—feeling that the self is not real
•: Distorted sense of environmental reality
•: Enhancement of empathy for others
•: Intensified sensations
•: Synesthesias —perceptions that stimuli can be perceived differently than normal, such as "hearing colors" or "seeing sounds"

Physical effects of LSD may include blurred vision, mydriasis, palpitations, sweating, and impaired coordination. Delusions and true hallucinations occur, but only rarely. Also rare when taking LSD are anxiety attacks, significant apprehensiveness, and states of panic. Many of the other hallucinogens cause nausea and vomiting. They all cause impaired judgment. An individual's responses to a hallucinogen are based on factors that include expectations of what will happen, the location where the drug is taken, and the ability to cope with the perceptual distortions.

Mescaline and psilocybin are more likely to cause hallucinations than LSD. When these occur, they usually end quickly if they are treated appropriately, in a setting that is secure. Some individuals, especially with LSD, continue to be disturbed, and a persistent psychotic state may remain. It is not fully known if the use of a hallucinogen has precipitated or revealed any preexisting psychosis, or if a hallucinogen can cause psychosis in a previously stable individual. Long-term or repeat users—especially of LSD—can experience drug effects long after stopping the drug. Sometimes, these effects occur after only one or a few uses. The recurrent episodes are referred to as flashbacks or as hallucinogen persisting perception disorder. Visual illusions are most common, but there can also be various hallucinations, alterations of self-image, and altered perceptions of space or time. Flashbacks can be precipitated by the use of alcohol, marijuana, or barbiturates. They are also precipitated by fatigue or stress, and can even occur with no obvious reason. The mechanisms of flashbacks are unknown, but they usually subside within 6–12 months. Rarely, they recur for years.

Epidemiology

The use of hallucinogens was highest in the 1960s, but there was a resurgence in the 1990s—especially in the high school and college-aged populations. In 2019, the prevalence of 1-year hallucinogen use was only 0.62% of the overall population in the United States. The prevalence of lifetime hallucinogen use was 9.32%. Hallucinogen use disorder is relatively uncommon, with a low risk of development following exposure to hallucinogens. The 2012 Global Drug Survey showed hallucinogen used to average between 5% and 8.9% of the population in most countries that were surveyed. Usage of hallucinogens is basically even between males and females, and there is no racial or ethnic predilection. The only racial significance is that many Native Americans used various hallucinogenic plants as part of ceremonies or rituals. Examples of these plants include peyote, maiden's acacia, and saguaro (see Fig. 16.6). Other plants with hallucinogenic properties include genista, mescal beans, nutmeg, sweet flag, and toloache.

Pathophysiology

The pathophysiology of LSD, psilocybin, and many of the designer hallucinogens is based on these drugs being serotonin receptor agonists. Mescaline differs because it is a phenylethylamine that is similar to amphetamines, with a mechanism of action that is not fully understood. LSD is taken orally, from blotter paper that the drug has been dropped onto in liquid form (see Fig. 16.7), or as tablets. The onset of action is 30–60 minutes, but the effects can last from 12 to 24 hours. A high degree of tolerance develops and disappears quickly. Mescaline is ingested orally as peyote buttons, with the onset of effects within 30–90 minutes, lasting for about 12 hours. Psilocybin is also taken orally, with effects lasting 4–6 hours in most cases. When a person smokes the drug known as N,N-Dimethyltryptamine (DMT), the onset is within 2–5 minutes, and the duration of effects is 20–60 minutes. Because of this shorter time period than the other hallucinogens, DMT has the street name of businessman's lunch. Users that are tolerant to any of these hallucinogens are also cross-tolerant to other drugs. Psychologic dependence is widely varied. There is no documented evidence of any physical dependence or withdrawal syndrome.

Etiology and risk factors

The reasons that people choose to take hallucinogenic drugs are varied, but primarily because of a wish to experience altered perceptions, feelings, and thoughts. They are also used in spiritual gatherings or celebrations by certain cultures, or to inspire artistic creativity. However, excessive use results in hallucinogen use disorder, which can have severe and sometimes permanent effects. The disorder is able to cause the death of the individual, such as when a hallucinogen is combined with another drug and results in judgment being impaired to such a degree that the person attempts to "fly" off of a rooftop. Risk factors for hallucinogen use disorder include genetics, personal history of substance abuse, family history of substance abuse, easy access to hallucinogens, younger age, and personal or family history of mental illness.

Diagnosis

Diagnosis of hallucinogen use disorder is usually clinical. Drug levels are not measured. Most hallucinogens, except for PCP, are not included in routine urine drug screens. The diagnosis of hallucinogen use disorder is the same as for the other disorders in this chapter: increased quantities of the drug, inability to stop taking the drug, negative effects on lifestyle, inability to perform life activities, giving up normal activities for the drug, dangerous use of the drug, and the development of psychological and physical problems as a result of usage.

Differential diagnosis

The differential diagnoses for hallucinogen use disorder include alcohol-related psychosis, bipolar disorder, schizoaffective disorder, schizophrenia, and schizophreniform disorder, delirium, delirium tremens, delusional disorder, Eastern equine encephalitis, hyperthyroidism and thyrotoxicosis, panic disorder, posttraumatic stress disorder.

Treatment

Acute hallucinogen intoxication requires treatment in a quiet and calm environment. The individual must be reassured that the effects of the drug are temporary but expected to occur—this usually calms the user down without any additional treatment. Anxiolytics are used if there is severe anxiety and include diazepam and lorazepam. Appropriate psychiatric care is required for persistent psychotic states or other mental disorders. No special treatment is needed for transient flashbacks, or for flashbacks that are not severely distressing. Any flashbacks related to anxiety or depression may require anxiolytics, however. For hallucinogen use disorder, treatment combines medications with drug rehabilitations therapy, cognitive-behavioral therapy, and psychosocial therapy.

Prognosis

The prognosis for hallucination use disorder is generally good when the individual follows treatment guidelines. However, some people relapse and return to using hallucinogens, resulting in a poorer prognosis. Also, for those who continue to experience flashbacks, the prognosis is worsened.

Significant point

A 2019 study found that the transition rate from a diagnosis of hallucinogen-induced psychosis to that of schizophrenia was 26%. This is lower than for cannabis-induced psychosis (34%) but higher than for amphetamines (22%), opioids (12%), alcohol (10%), and sedatives (9%).

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B978032385837300008X