Recently, the small editor was assigned by the company to be responsible for the evaluation of the effect class. Because the judgment and annotation of the data results of the effect class evaluation are subjective, the machines and algorithms cannot make correct judgments, and manual participation is required. Therefore, the work related to data annotation is involved. Because I have not done similar work before, there are many unexpected problems. Now I want to share them with you. I hope you will avoid detours.
1. There are various user data. The demander does not look at the real user data. The labeling rules provided are particularly broad. During the labeling process, there are many problems not involved in the rules, and the cost of rule communication is high;
2. The requirements that the demander needs to evaluate belong to the experimental function, which leads to constant changes in the rules. Sometimes the rules of the first day are completely opposite to those of the second day, which wastes human resources;
3. The demander provided less auxiliary information, and communicated when problems were found in the later marking process, which increased the cost of human communication;
4. The annotation personnel are not of testing background, lack of functional experience of annotation data, and have a shallow understanding of the logic behind the function, leading to deviation in the understanding of annotation rules, resulting in low accuracy;
5. The marking task is urgent, and the marking is started without deep understanding of the marking rules, resulting in low accuracy of marking;
6. For some labeling tasks with strong logic, the accuracy rate of labeling is very low;
7. The trial marking is carried out by the marking personnel, because their understanding ability and awareness of the evaluation function are not enough, resulting in fewer problems found, leading to some problems only exposed when reviewing the data.
In view of the above problems, mainly from three aspects to improve. On the one hand, it is to improve the ability of labeling personnel; on the other hand, it is to reduce unnecessary labor costs by optimizing the process. The third aspect is to use tools to assist the dimensioners in dimensioning.
1. Ability improvement of marking personnel
1） For specific annotation tasks, each annotation personnel analyzes the specific reasons for the wrong annotation data, and for different reasons "Suit the remedy to the case";
2） Increase the assessment, force the labeling personnel to experience the input method function, and improve the understanding of the labeling function;
3） Sort out labeling rules and problems encountered, and train labeling personnel;
4） Draw a logic judgment flow chart for the marked tasks, and assist the marking personnel to understand the marking rules through the flow chart;
2. Process optimization
1） Add the submission process of labeling requirements, including function introduction, accuracy requirements and other information;
2） Before formal intervention in marking, add trial marking, and supplement marking rules in advance if problems are found through trial marking;
3） The person in charge of the trial marking also participated to make up for the problems that the marking personnel could not find;
4） Add marking personnel to enter the marking process. If the marking accuracy rate is not up to the standard, formal marking cannot be carried out, and training will be carried out until the accuracy rate is up to the standard;
5） The demander shall provide at least 100 marked cases;
3. Tool aided dimensioning
1） Develop relevant tools to pre label parts, assist the labeling personnel, and improve the efficiency and accuracy of labeling;
2） The research data annotation platform starts to review the data during the annotation process to find problems early and improve efficiency.
These are some problems and solutions for data annotation in effect evaluation. If you have better suggestions or schemes, please leave a message for discussion and exchange.
On the dream chasing trip, Xiao Bian has been on the road, hoping to go with you and grow up together. Sogou Test welcomes you to share your experience and help more people.
