The "Fresh Look" essays are an attempt to look at Life through the eyes of an engineer and as you read through them you will now and then find reference to various concepts. This section is an attempt to explain these concepts.
Think of a mountain water reservoir with an exit pipe leading down into the valley. The end of the pipe is closed with a valve. You slightly open the valve and water starts to jet out several feet. You open it a bit more and even more water squirts out. Open the valve completely and you have a raging torrent.
The valve you are turning is acting as an Amplifier - little effort: big result. And notice that the result is proportional to the effort, more effort: more result. Or as we say, the output is proportional to the input. The ratio of the work you do opening the valve to the work the water does when it squirts out is called the Gain of the amplifier.
Imagine the valve is rather difficult to turn. With a little ingenuity you can use some of the water coming out of the pipe to drive a turbine which you use to help you to open the valve. This will make operation of the valve much easier. You need to do less work to control the valve. In the jargon, you have added some of the output of the amplifier to the input of the amplifier in order to help or reinforce the input. Less input for the same output means the gain of the amplifier has been increased. When some of the output is fed-back to reinforce the input, we say we have Positive Feedback.
If you feel intuitively that the application can be a bit tricky, you are right. As you increase the amount of positive feedback, the amplifier gain increases until a point arrives when almost no input is sufficient to drive the amplifier to maximum output. At this point the amplifier is extremely unstable. Some little glitch on the input causes an output which is fed back to increase the input, which gives more output which still more increases the input, until the amplifier is hard on with no input. Having an output for no input means the gain has become infinite. The Amplifier has become a signal Generator. (The type of signal it generates depends a lot on how the signal gets from output to input - on what the feedback path is like).
Going back to the valve and pipe analogy again, let us now connect the turbine in such a way that it opposes you when you turn the valve. In other words it makes the valve harder to turn and reduces the gain. A fairly pointless exercise you would think. But in fact most amplifiers have more than enough gain - their problem is that they have too much gain and that this gain is unstable, varying with time, temperature etc. Many amplifiers are also non-linear ie. the output is not proportional to the input. (If your hi-fi amplifier were non-linear its output would sound distorted). Feeding back some of the output in such a way as to oppose the input is called Negative Feedback and is a wonderful way to reduce the gain of an amplifier to reasonable values and at the same time improve its stability and linearity.
Feedback in real life
The effect of positive feedback can be seen when a salesman watches a customer to see which features please him and then feeds this back to him, amplifying its effect. A more dramatic example of positive feedback is to be seen in a riot where everyone receives (and transmits) excitement-increasing signals. An actor in front of an approving audience is receiving positive feedback. He becomes "high", which can make him give a better performance, which makes the audience approve more, which ...
Negative feedback can be seen in the action of the Devil's Advocate - a person who damps down the enthusiasm of a highly-creative and somewhat unstable character, pointing out all the faults in his ideas. The combination of Enthusiast and Devil's Advocate exists in us all and enables us to progress without excess.
There are many interesting examples of feedback in Nature. You see a strange object on the ground and lean forward to pick it up. How much force should you apply? If you give a light tug and it is heavy you may raise it a bit but then it will slip out of your grasp: if you give a heave and it is very light you may fall on your back. So you give it first a little exploratory tug and see what happens. What you are doing is weighing it by first measuring the strain in your arm muscles - and then you switch on the main power and lift. But if it is a well-known object you don't spend a lot of time measuring its weight before you pick it up. In the middle of a tennis match you pick up the tennis ball rapidly because you "feed-forward" the weight information to your arm muscles (and also the "correct grip" information to your hand). The amusing effect of wrong information being "fed-forward" can be imagined if a tennis ball filled with lead were left on the tennis court.
The importance of getting the feedback right
In the sixties, everyone in the aeronautic world was building vertical take-off aircraft. The Germans were no exception and they built one in Munich, called the VJ-101. For propulsion the VJ-101 had two large rotatable jet engines, one at each wing-tip, which could be pointed downwards for hovering. There was also a small fixed downward-pointing engine in the fuselage which gave extra thrust during hovering and by giving it more or less fuel than the main jet engines, it could be used to control the pitch axis (ie. nose up, nose down). Now hovering is tricky in a vertical take-off aircraft. If the aircraft starts to roll, say clockwise, you have to feed some more fuel to the right-hand engine and less to the left-hand engine. This will stop the roll but if you want to get back level, you have to roll back in the opposite direction and when in the level position give a burst of fuel in the opposite direction to stop more rolling. Like a space-craft. Of course you have to do this on the pitch axis as well, and it's a good idea to keep your eye on the height above ground so you need to be able to control the fuel supply to all three engines. Not surprisingly, it was thought too much for the pilot to do this all at once. So to help him, the fuel supply to the wing-tip engines was modified by a gyroscope, in such a way that if the aircraft started rolling, fuel would be given to the two engines so as to stop it rolling - negative feedback. "All" the pilot had to do during hovering was to keep the aircraft level in pitch, and control the height above the ground (apart from all the other things a test-pilot does in a prototype aircraft!)
Well, one day the VJ-101 blasted off down the runway on a Short Take-Off test. (By having the wing-tip engines point down at 45 instead of 90 degrees, the VJ-101 would take off in about 100m. Advantage - great saving in fuel as opposed to true vertical take-off). Imagine the surprise and horror of all concerned when the VJ-101, immediately after take-off and still only 50m up, was seen to start to roll. It did a complete roll and had started on another when the pilot understandably lost his nerve and pulled the ejector. He made it, (the parachute just had time to open) but the VJ-101 was a write-off.
The explanation was found by studying the telemetry records (data radioed from the aircraft). At first, level flight so no correction signal from the gyroscopes. Then a buffet of wind tipped down the port wing. The gyroscope unaccountably fed less fuel to the port engine and more to the starboard engine. Aircraft rolls a bit more. Increased angle of roll detected by the gyroscope which feeds yet more fuel in the wrong direction. Aircraft continues rolling at an accelerating rate. Now you cannot roll an aircraft at just over its stalling speed: lift disappears and it will just fall out of the sky, which is what happened.
What had occurred, and was confirmed later by examination of the wreckage, was a very simple wiring error - the leads to the gyroscope had been accidentally connected the wrong way round. What was supposed to be Negative Feedback had become Positive Feedback!
The only surviving VJ-101 can today be seen in the Deutsches Museum in Munich.
Let us take a normal public telephone as an example of a communication link. The first thing an engineer would want to do is to find its "bandwidth", or the range of frequencies it could pass. A quick check would be to whistle into the microphone starting at a very low frequency and gradually rising. He would then ask the person at the other end what had he heard and he would probably say "I didn't hear anything until you got to 100Hz then it was OK until you got to 3500Hz and then I couldn't hear anything more". Hz means cycles per second. The engineer would say the overall Bandwidth of the telephone link was from 100 to 3500Hz or 3400Hz. Which is OK for a telephone as it corresponds to the loudest part of the human voice when speaking normally.
Bandwidth is the rate at which a signal can be handled, the rate at which data can be transferred. Indian smoke signals, flag semaphore, and deaf and dumb language are low bandwidth signals. Morse code is wider (higher signal rate) but still narrower than speech. A good hi-fi has a signal bandwidth about 7 times wider than a telephone. And so on up to satellite links which have signal bandwidth in the hundreds of megaherz.
Human beings can also be studied using the concept of bandwidth. Our ears for instance, have a bandwidth of from 20 to 18kHz. We can move the lighter parts of our body in 1/10 sec. which corresponds to a bandwidth of 3Hz. There are limits to the rate at which we can track a moving object in three dimensions, react to some stimulus, type, follow a conversation, talk. Our brains are not infinitely fast, we can only make a certain number of decisions per second. As is to be expected of a very complex mechanism, humans have bandwidths which can be varied to optimise performance under different conditions. Asked a question, we can give a rough answer quickly, but require more time for a considered answer, when the slow moving (narrow bandwidth) brain has had time to work on the problem.
Put a child on a swing and give a push. The swing will go back and forth at its "natural" frequency with ever decreasing amplitude until after a certain number of swings it comes to rest. The swing is an "oscillator". Another oscillator is a church bell. Neither of these oscillators goes on for ever after the initial push or strike. This is because they lose a little of the energy (by friction somewhere) at each swing until there is none left. The child learns that it is possible to top up the energy lost by swinging its legs at a special frequency. This frequency is the "natural" frequency of the swing. The child also finds that by pushing energetically it is possible to more than compensate for the energy lost per swing. The amplitude of the swing then builds up until it is limited by the suspension.
The child's kicking frequency is said to be "in resonance" with the natural frequency of the swing and the child has found the "resonant frequency".
You can get the same effect if you go into your bathroom and whistle up and down the scale. At one frequency the amplitude will suddenly increase. You have found the resonant frequency of the bathroom. People sing in bathrooms because it magnifies their voice, makes it sound more "resonant".
Notice that the bathroom resonant frequency is quite well defined, you really have to hit the right note. In technical language we say it has a high "Q" or Magnification Factor. Now hang some towels around and try again. The bathroom still resonates but it is not so sensitive to one frequency and also the effect is not so pronounced. It is more "flat tuned". You have reduced the Q. The wall and ceiling of concert halls are covered with sound absorbing material to avoid resonances, to reduce Q, to make the hall flat tuned so that no note is falsely emphasised.
Another way of looking at the bathroom in the example above is to think of it as something which is particularly sensitive to one frequency over all others. Put in (sing) a lot of frequencies and it would filter out just one - its resonant frequency. It is acting like a filter.
Filters are much used in communications.
In electronics, a filter is a box fitted with an input and an output connector. Inside the box are one or a number of circuits which act like oscillators. They usually have different resonant frequencies and different Qs. Printed o the side of the box is what it does. If you put a "440Hz pass filter" between your record player and the loudspeaker and then play a record you would only hear when some instrument was playing the note A above middle C. All the rest would be silence. Now that would be a fairly "narrow" "pass" (high Q) filter. It is possible to build a filter, which would pass all the notes from the G below A to the B above A ie. from 392 to 494Hz. This filter would have a rather "wider" "pass-band" (lower Q) than the first filter.
It is also possible to build "reject" filters which do the opposite of the above eg. pass everything except the frequencies from 392 to 494Hz.
We can construct "reject" filters in our head which push familiar sounds into the background and allow us to concentrate on new, interesting information. In a quiet room you soon forget the ticking of the grandfather clock - until it stops. Our brains must be full of constructs which are similar to "reject" filters. When you consider the flood of data arriving every second through our senses, we must filter out most of it.
All the filters so far are "fixed" tuned. There are "tuneable" filters, and not only tuneable but there are "variable band-width" (or variable Q) filters.
There are filters which can be shaped so they give maximum output from some complex sound (like a clarinet playing middle C). Listen to an orchestra through such a filter and hear the sound of the clarinet boom out each time it hits middle C. This is called a "matched filter".
Filters could be built to follow a sequence of events eg. to listen to an orchestra playing Beethoven's 9th and give an output only if someone is playing "God Save the Queen" on a mouth organ in the background. We build filters like this in our head very easily. Pass filters have generally been constructed in our brain to emphasise something that is important or gives pleasure. We can all hear if our name is mentioned in a noisy crowded room. Playing an initially difficult piece of music over and over again will gradually tune a filter to it until you begin to appreciate it. The same for "dry" wines, which most people don't like the first time. "Falling in love" must surely be the act of tuning yourself to "her". The more you finely tune the filter, the greater pleasure you will receive. But this has its dangers. Finely tuning yourself (to receive more pleasure) to one of her moods may only be done at the expense of relative displeasure during her other moods.
When we say "I got used to it" (and it is truly amazing what humans can get used to) we are really saying that we have constructed a complex group of reject and pass filters to remove from our consciousness what we don't like and emphasise what we do. An important measure of intelligence must surely be the ability to quickly change these filters to match a new environment.
Let us return to the engineer testing the telephone. Having measured the overall bandwidth he would now simply listen to the earpiece with no one talking and measure the strength of the clicks, crackles, and hissing noises that occur as background to a telephone conversation. This he would call the "Noise Level".
Noise can come from many sources - bad connections, nearby power lines, faulty components, or just noise getting into your ear from an airplane flying overhead. In real life it can take many forms: a bad photocopy, sun shining on the TV screen, the shimmering earth's atmosphere that distorts astronomers' star images, coughing in a concert hall, diesel fumes in a scented flower garden, a wine that has "gone off". Noise is any disturbing signal that reduces the quality of the signal you want.
Signal to Noise Ratio
Our engineer, having measured the noise level, would now
ask the person at the other end to send a constant signal so he could make an estimate of the relative loudness of the noise and the signal. He would call this "Signal to Noise Ratio" and it should obviously be as high as possible. As you can see from the noise examples given above, noise is always with us. The only thing we can do is to make its influence as small as possible, by having as high a signal to noise ratio as possible. This can be done in one of two ways: 1.either increase the signal (shout louder at a noisy cocktail party), read books with large print (if your eyes are failing) or: 2. reduce noise somehow (only use pure components in your cooking, take your telescope to a mountain peak so there is less trembling air between you and the star, draw a curtain in front of the window to stop sun shining on the TV screen).
Use of filters to improve signal to noise ratio
In a communication link, filters are often used to remove noise while letting the signal pass through. They improve the signal to noise ratio. Say we are trying to listen to a weak Morse code signal at 1kHz on a radio with lots of noise in the background. Our ears have a bandwidth of from 20Hz to 18kHz and so a lot of unwanted and unnecessary noise in this band is getting through and distracting us. The situation can be greatly improved if we put a 1kHz pass filter on the output of the radio. This will let the signal pass through undisturbed but block most of the noise. Any noise at 1kHz will still get through, of course.
In Life the type of filter used to improve the signal to noise ratio can take many forms. You can "direct your attention" to something, as you do when you wait for someone at an airport. You construct a pass filter in your head which corresponds to his face. All other faces are filtered out.
More about noise
Noise is funny stuff. It represents the fundamental randomness of Nature and was first studied in high-gain amplifiers where it appeared as a hissing sound. (Hence the word "noise" for all disturbing signals). It is actually produced by random movements of electrons in the amplifier itself and limits the minimum signal that can be amplified.
Now imagine we have a collection of narrow pass-band filters. Number 1 allows through frequencies from 1-2Hz, number 2 from 2-3Hz, etc. going up to the last one which passes from 10 000 to 10 001Hz. We wire them all up to the output of a high-gain amplifier. So as not to complicate the issue at the moment, we will assume the amplifier can amplify all frequencies - it has an infinite bandwidth. Let's short the input out and turn up the gain. Out comes the rushing sound of its self-generated internal noise. Now let us listen to the output of each individual filter. To our surprise the amount of noise from each filter would be the same.
Sure, it would sound different from each filter but the actual amount (power) would be the same. From the 1000-1001Hz filter you would hear a slowly wobbling 1000Hz whistle. From the 100-101Hz filter you would hear a slowly wobbling hum of around100Hz. For a 100Hz filter to have an output, it must have a 100Hz input. And the same for all the other filters. Conclusion - noise contains all frequencies.
Noise is a wide-band signal.
This is a very important fact.
Uses of noise
There are many practical applications of this. We can put wide-band noise into a communication link and measure what frequencies come out. This will tell us the bandwidth of the link.
The complex mechanical structure of an airplane can be looked on as a collection of resonators or filters. One way to test it is to first fit vibration sensors to all the important parts (wings, control surfaces etc.) and then fly around in turbulent air. The turbulence acts like a noise signal input to all these filters. By analyzing the frequencies coming out of the vibration sensors it is possible to measure the resonant frequency of the various components and determine how stable the aircraft is.
Noise into a filter
Imagine we have built a complex filter that only passes the frequencies present in the output of an oboe playing middle C. If we feed noise into this filter and listen to the output, to your amazement you will hear the sound of an oboe playing middle C! This is because noise contains all frequencies and our filter has just selected out those corresponding to an oboe playing middle C.
This is a very important effect and explains a lot of mysterious phenomena.
We can build filters in our head. Look at the stars at night, a badly plastered wall, tea leaves in a cup, the embers of a fire, and you will see pictures. These pictures are the output of a matched filter you have just built, filtering out the "cat's head" from all the other possible images in the noiselike input.
More ways of increasing signal to noise ratio.
To recapitulate. We can get a message through in a noisy environment by increasing the signal to noise ratio by:
- increasing the amplitude of the signal,
- decreasing the noise with a noise filter,
- or both.
Alternatively we can use the time element and take longer over sending the message.
Remember that even in the presence of noise some signal will get through. So another way is to repeat the message more than once, or send it in a different way, or parts of it more than once. This is called adding "Redundancy". The idea is that if part of the message is missed you get it the next time around or it can be reconstructed from what you did receive.
Perhaps because of the generally noisy level of Life, or the inattentiveness of the average listener (noise in his head), our speech is on purpose very redundant. We repeat ourselves, by making little puns and adding picturesque allusions.
Redundancy is most easily recognized when we have to pay for it. For example, if each word in a telegram costs $1, it is remarkable how we are able to shrink an ordinarily long message by squeezing out all the redundancy. Of course, the other side of the coin is that each word must be received perfectly or noise free. No errors can be tolerated.
Noise reduction by feedback
This requires a transmitter and receiver at each end of the communication link. The idea is that the receiver confirms reception of the message in some way. The Navy use feedback as in:
Captain `Steer 090'
Seaman `Aye, aye Sir. Steer 090'
Captain `Full ahead both'
Seaman `Aye, aye Sir. Full ahead both'.
But we use feedback more subtly. We send out the first part of our message and then wait for the feedback. The opening is usually routine:
John `Ah, Peter'
The link has been established and now data starts to flow. If Peter sends back the "right" signals, gasp of incredulity, grunt of confirmation etc., John knows the message has been correctly interpreted and sends along the next item. But if he receives a blank stare, or something like `I haven't the faintest idea what you're talking about' he obviously needs to rephrase his first remark, to expand it, add redundancy. On the other hand, an impatient `Yes, yes' would cause John to strip out a lot of the redundancy he was planning to add to the next part of the message.
Feedback is very necessary as you will soon find if you try to talk to someone wearing a funny mask at a noisy party. You will soon feel you are talking to yourself and walk off in disgust.
Lack of feedback is also the reason given by some of my friends for not leaving a message on my telephone answering machine. When the time comes for them to speak their message they cannot talk to a "dead microphone".
Let us now can take another look at the famous formula which Claude Shannon of the Bell Telephone Company discovered around 1948, and which I quoted in the introductory essay "A Fresh Look". I show it again:
C = W log 2 (1+S/N)
C = the channel capacity in “bits” per second
W = channel bandwidth in Herz (or cycles per second)
S/N = signal to noise ratio
You are driving down a road and come to a fork. Go left or right? Your passenger, who has a street-plan on his knees says "right". You have received one "bit" of data. A bit is the smallest "piece" of data that exists. You had a choice of two roads and no knowledge of which to take. One bit of data was all you needed to remove the uncertainty of which direction to take.
Information vs. data
"The receipt of information in the form of a message implies uncertainty in the mind of the recipient before the message arrives. A measure of the information content of a message can thus be based on the amount of uncertainty it has removed".
Note that "information" is "news", it is data you didn't have. If your passenger is grumbling about the weather conditions, you are receiving lots of data but very little information - you can see the weather is bad yourself and there isn't much uncertainty about the reason for your passenger's annoyance; you were both going to play tennis.
Information is therefore also measured in bits. If I have to go through a town and there are ten left/right choices, and each one is a "toss-up" for me, I am going to need 10 bits of information in order to navigate the town successfully.
As you are driving through an unknown town your navigator/passenger is constantly feeding you "bits" of information, left, left, right etc., perhaps at the rate of one a second. He is feeding you information at the rate of 1 bit per second and obviously needs a "channel capacity" (speech channel from his mouth to your ear) capable of passing at least 1 bit per second. The Shannon formula can be used to tell you the maximum noise that can be tolerated for you to hear the message clearly.
Use of the Shannon formula
When the Shannon formula appeared it was immediately seen as very important for several reasons:
- It showed that there were two factors which determined how quickly a message could be sent over a given channel – the signal bandwidth and the signal to noise ratio. This was by no means new, everyone knows this intuitively once the meaning of the words "bandwidth" and "signal to noise ratio” have been explained to them (as I hope I have to you), but it was deeply satisfying to have a theoretical basis for this knowledge.
- It meant that as it was now possible to measure data, it was now possible to measure information (information meaning data that removed uncertainty). As an example - how much information do you need to find a given name in the London telephone directory? To work this out you ask the person who knows the name "Is it in the first half?". He will say "yes" or "no". If he said "yes", it is in the first half, now you ask him "Is it in the first half of the first half?" And so on, dividing the last page into top half and bottom half etc. until finally you ask him if it is the top name or the bottom name. If there are 8 million names in the book you will need 22 bits of information from him (22 yes/no answers) in order to find the given name.
As information could now be seen to be (and measured) as a series of yes/no questions, it started everyone thinking about information generally and investigating (for instance) the information handling capacity of a human being, how much information you received from a news bulletin, how much information is required to bake a cake, what is the channel capacity of the nerve connecting the brain to a finger, how many bits per second can the eye receive etc. etc.
As far as I know, no one has yet used Information Theory to study Art. Which is what started me writing these essays, you will remember.