The amount of information refers to the measurement of the amount of information.1928 R5. L. Hartley first put forward the preliminary idea of information quantification. He defined the logarithm of the number of messages as the amount of information.If there are m kinds of messages in the source, and each message is likely to be generated equally, then the information amount of the source can be expressed as I=logm.However, the amount of informationsystem study , or from 1948 CE.Shannon The groundbreaking work began.stayinformation theoryThe message output by the source is considered to be random.That is, before a message is received, you cannot be sure what message the source is sending.The purpose of communication is to enable the receiver to remove as many doubts (uncertainties) about the source as possible after receiving the message. Therefore, the removed uncertainty is actually the amount of information to be transmitted in communication.
In 1928, R.V.L. Hartley put forward the preliminary idea of information quantification. He defined the logarithm of sign value m as information quantity, that is, I=logtwom。C.E. Shannon, the founder of information theory, made a thorough and systematic study of information quantity.In 1948, Shannon pointed out that the symbol given by the source is random, and the information quantity of the source should be a function of probability, expressed by the information entropy of the source, that is
, where PiIndicates the probability of different types of symbols of the source, i=1, 2,..., n.
For example, if a continuous source is quantized into four layers with equal probability, that is, four symbols.The amount of information given by each symbol of this source shall be
, and Hartley formula I=logtwom=logtwo4=2bit is consistent.In fact, Hartley formula is a special case of Shannon formula with equal probability.
Figure 1
Basic content The actual source is mostly a memory sequence source. Only after mastering the probability characteristics of all sequences can we calculate the entropy H of an average symbol in the sourceL(U) (L is the sign number, which is usually difficult. If the sequence source is simplified to a simple first-order, homogeneous, ergodic Markov chain, it is relatively simple. According to the conditional probability of the sign Pji(i.e. the probability that the previous symbol is i and the next symbol is j), the stability probability P of the traversing source can be calculatedi, and then PiAnd PjiFind HL（U）。See Figure 1.
Where H (U | V) is called conditional entropy, that is, the uncertainty of the following symbol U when the previous symbol V is known.
There is a conceptual difference between information quantity and information entropy.Before receiving the symbol, it is uncertain what symbol the source is sending. The purpose of communication is to enable the receiver to remove the doubt (uncertainty) of the source after receiving the symbol, so that the uncertainty becomes zero.This indicates that the amount of information the receiver obtains from the sender's source is a relative amount (H (U) - 0).Information entropy is a physical quantity that describes the statistical characteristics of the source itself. It represents the average uncertainty of the symbol generated by the source. It is always an objective quantity whether there is a receiver or not.
Obtain the information of another symbol u from a symbol V in the source
The quantity can be expressed by mutual information, that is
I（U；V）= H（U）-H（U|V）
It indicates that there is still doubt (uncertainty) about the source symbol U after receiving V.Normally
I（U；V）≤H（U）
That is, the amount of information obtained is smaller than the information entropy given by the source.
Continuous information sources can have unlimited values, and the amount of output information is infinite, but mutual information is the difference between two entropy values, which is a relative quantity.In this way, no matter the continuous or discrete source, the amount of information obtained by the receiver still retains all the characteristics of the information and is limited.
With the introduction of information, communication, information and related disciplines can be established on the basis of quantitative analysis, which guarantees the establishment and development of relevant theories[2]。
brief introduction
Announce
edit
The so-called amount of information refers to the information measure or content required to select an event from N equal possible events, that is, the minimum number of times to ask "yes or no" in the process of identifying a specific event among N events
Shannon （C. E. Shannon）information theoryapplicationprobabilityTo describe uncertainty.Information is defined by uncertainty measures. The less likely a message is, the more information it has;The more likely the message is, the less information it containsprobabilitySmall, the more uncertainty, the more information, and vice versa.
Modern definition of information。[2006, Medical Information (Journal), Deng Yu, etc.]
Information isSubstanceenergy, InformationAndIdentification of attributes。inverseWienerInformation definition
Information isIncrease in certainty。inverseShannon Information definition
Information isthingPhenomenon andAttribute IDCollection of.2002
staymathematicsThe transmitted message isprobabilityMonotone descent function of the.If you select a number from 64 numbers and ask: "Is it greater than 32?", half of the possible events will be eliminated regardless of whether the answer is yes or not. If you continue to ask such questions six times, you can select a number from 64 numbers.We can use 6 bits of binary to record this process, and then we can get this information.
A measure of the amount of information.1928R. V.L. HartleyFirst, he put forward the preliminary idea of information quantification. He defined the logarithm of the number of messages as the amount of information.If the source hasmIf each message is generated equally, the information quantity of the source can be expressed asI=logm。However, the amount of informationsystem study From 1948C. E. ShannonThe groundbreaking work began.
The statistical characteristics of information are described as early as 1948Shannon Compare the concept of entropy in thermodynamics withPrinciple of entropy increaseThe result of introducing information theory.Advance investigationPrinciple of entropy increase。ThermodynamicPrinciple of entropy increaseIt is expressed as follows: there is a state function entropy, onlyIrreversible processCan makeIsolated systemThe entropy of the isolated system increases, while the reversible process does not change the entropy of the isolated system.It can be seen that: first, entropy and entropy increase are system behaviors;2、 This system isIsolated system；3、 Entropy is statisticalState quantity, entropy increase is statisticalProcess quantity。When discussing the entropy expression of information, we should pay full attention to the existence of these characteristics.And we know that the information propagation in a given system is an irreversible process.
E.H.Weber
stayinformation theoryThe message output by the source is considered to be random.That is, before a message is received, you cannot be sure what message the source is sending.The purpose of communication is to enable the receiver to remove as many doubts (uncertainties) about the source as possible after receiving the message. Therefore, the removed uncertainty is actually the amount of information to be transmitted in communication.Therefore, the amount of information received is equal to that of the source in numerical value when there is no interferenceInformation entropy, whereP（xi) Get the number for the sourceiSymbolicprobability。But conceptually,Information entropyIt is different from the amount of information.Information entropyIt is a physical quantity that describes the statistical characteristics of the source itself.It is the average uncertainty of the source and an objective characterization of the statistical characteristics of the source.It always exists objectively whether there is a receiver or not.The amount of information is often targeted at the receiver. The so-called receiver has obtained information means that the receiver has released the average uncertainty of the source after receiving the message. It has relativity.The description of the amount of information must be introducedMutual informationThe concept of.
formula
stayinformation theoryMutual information is defined as:I(X；Y)=H(X)－H(X|Y), the latter term on the right of the number formula is called conditional entropy, which can be expressed for discrete messages, and it means knownYLater, yesXThe uncertainty that still exists.Therefore, mutual informationI(X;Y)Yes means when receivedYInformation source obtained laterXThe amount of information.AndMutual informationCorrespondence, often calledH(X）It is self information.Mutual informationIt has three basic properties.
① Non negative:I(X;Y）≥ 0, mutual information is 0 only when the received message is statistically independent of the sent message.
formula
② Mutual information is not greater than the entropy of information source:I(X;Y)≤H(X)That is, the information received by the receiver from the source must not be greater than the entropy of the source itself.They are equal only when the channel is noise free.
③ Symmetry:I(X;Y)=I(Y;X), i.eYImplicitXandXImplicitYThe mutual information of is equal.
For the mutual information of continuous information sources, it still represents the difference between two entropies, so it can also be directly generalized from the discrete case, and all the characteristics of the discrete case above are maintained, that is, the actual information source is a combination of single information sources, so the mutual information of the actual information sourceI(X;Y)The mutual information of the above single message can also be directlyI(X；Y)To promote, that isI(X;Y)=H(X)-H(X│Y)。
computing method
Announce
edit
information theoryfounderC.E.ShannonIn 1938, the concept of bit was first used: 1 (bit)=
The ith object in m object sets, the state set measured by n observation and control indicators
Full information TI=
。
It is known from the results after the test that the uncertainty before the test is reducedShannon The amount of information defined, i.e
Free information FI=- ∑ pi
，（i=1,2，…，n）。
Where, pi is the observation and control weight corresponding to the random variable xi, which tends to map the distribution of its actual stateprobability。The reduction of uncertainty before the test caused by its internal distribution is called prior information or predicate constraint information.The risk is hidden inrandom variableThe internal structural energy (that is, the effective energy that continues to play a role in the formation of this structure) has not changed.This function can be displayed and mapped
Constraint information BI=TI-FI.
The research shows that the ratio of m observation and control objects to n observation and control indicators for standardized controlPreferred order of comparative income, which is consistent with the preferred order of its free information FI;Moreover, the more free the observation and control objects are, the smaller the risk is;The constraint information BI is the intrinsic measure that maps its risk, namely risk entropy.
Describe the information asInformation entropy, YesState quantity, its existence is absolute;The amount of information is entropy increase, which isProcess quantity, is the quantity related to information dissemination behavior, and its existence is relative.In consideration of systematicnessStatisticOn the basis of, it is believed that the amount of information is determined by the specific source and specific destination range, and it is a statistic describing the potential flow value of information.This statement is consistent withPrinciple of entropy increaseRequired conditions:
1、 Composition of "specific source and destination range"Isolated systemThe amount of information is a system behavior rather than a single behavior of the source or sink.
2、 The information quantity is defined as statistics.This statement also shows that the amount of information does not depend on specific communication behavior, but is an evaluation of the potential flow value of an information "specific source and specific destination", rather than the information flow that has been achieved.Thus, the amount of information realizes the measurement of information[3]。
Calculation process
Announce
edit
How to calculate the amount of information?In daily life, rare events are easy to attract people's attention once they occur, while common events will not attract attention, that is to say, rare events bring a lot of information.If it is described in statistical termsprobabilitySmall events have more information.Therefore, the eventprobabilityThe smaller, the greater the amount of information.That is, the amount of information is related to the frequency of events (i.eprobabilitySize).
1. If the known event Xi has occurred, it indicates the amount of information contained or provided by Xi
H(Xi) = −
Example: If it is estimated that Xie Jun's probability of winning the championship in one chess match is 0.1 (recorded as event A), and her probability of winning the championship in another chess match is 0.9 (recorded as event B).How much information do you get from her when you know she won the championship?
H(A)=-
≈ 3.32 (bits)
H(B)=-
≈ 0.152 (bit)
2. The formula for calculating the amount of statistical information is:
Xi represents the ith state (n states in total);
P (Xi) - indicates that the ith state occursprobability；
H (X) - indicates the amount of information needed to eliminate the uncertainty of this thing.
Example: When a coin is thrown into the air, there are two possible states after landing: one is facing up, the other is facing up. Each state appearsprobabilityIs 1/2.Such as throwing evenlyRegular hexahedronThere are 6 possible states for each stateprobabilityBoth are 1/6.Try to compare the uncertainty of dice and coins by calculation.
H (coin)=- (2 × 1/2) ×
≈ 1 (bit)
H (dice)=- (1/6 × 6) ×
≈ 2.6 (bits)
Two inferences can be drawn from the above calculation:
[Inference 1] If and only if one P (Xi)=1 and the rest are equal to 0, H (X)=0.
[corollary 2] If and only if a certain P (Xi)=1/n, i=1,2,..., n, H (X) has a maximum log n.
developing process
Announce
edit
Now known asInformation society，Modern Information ScienceThe theory and its application pay great attention to the quantitative measurement of information.In the 1980s,britainFamous information scientistB. C. Brooks, when elaborating the process of human information (intelligence) acquisition, the reception process of sensory information was deeply studied, andPerspective principle-- The observation length Z of the object is inversely proportional to the physical distance X from the observer to the observed object. Introducing information science, Z is proposed=
OflogarithmHypothesis.It can be used to better illustrate that in information transmission, information changes with time, spacesubject(industry)Logarithmic transformation。However, with regard to the information search behavior of users, the conclusion that "the proportion of obtaining information from the nearest is the highest, and the proportion of obtaining information from the farthest is the lowest" requires new theories for new generalization when it comes to cross domain integration and the existence of the Internet.logarithmPerspective transformation, from experimentPsychophysics。1846GermanypsychologistE.H.WeberProposedDuke WeberFormula: △ I/I=k.Here, △ I represents just perceptible difference threshold, I represents standard stimulus physical quantity, and k is a constant less than 1.Later, Fechner called this rule about the threshold of differenceWeber's lawIn 1860, he put forward the famous FekennaLaw of logarithm: Psychological sensation value S is physicalStimulus quantityI'sLogarithmic function, that is, S=cLogI, and c is the constant determined by the special feeling mode.
In 1957, Stevens proposedPower law: S=bIa, a and b are characteristic constants.Psychophysical functionWhat is obediencePower lawOr obeyLaw of logarithm？W. S. Togerson believes that this cannot be solved through experiments, but is a problem of making choices in experiments.G. Ekman assumes that Fechner'sLaw of logarithmIt is generally correct to deducePower lawIt is a special case of the law of logarithm.
ChinaScientists with outstanding contributionsCheng Shiquan, published in 1990Fuzzy Decision AnalysisIn a book, review and quotationYu HongyiFor "systematic qualitative and quantitative transformation, a convenient, feasible, scientific and reliable method for qualitative sequencing and quantitative transformation" was summarized.Yu HongyiBy using the explicit frequency information and the potential general order information - weights, the fuzzy system can be easily and effectively transformed into a clear engineering system.Its measurement mode is:
F(I)=Ln(max{I}-I+2)/Ln(max{I}+1)。
Where, I is the sequence number of the object in question according to a certain index, and F (I) is its membership degree.The ingenious use of "automatic interlocking" mechanism in practical application is really simple, practical and effective.The so-called "automatic chain" mechanism is that "the evaluator cannot fail to show himself or be evaluated while evaluating other people's affairs"[4]。
