Flink job Keep restarting

Abnormal performance

For the same business, four jobs are processed separately, but one of them is abnormal and restarted continuously, as shown below:

Troubleshooting process

View the service log as follows:

 org.apache.flink.streaming.runtime.tasks.AsynchronousException: Caught exception while processing timer.     at org.apache.flink.streaming.runtime.tasks. StreamTask$StreamTaskAsyncExceptionHandler.handleAsyncException(StreamTask.java:1117)     at org.apache.flink.streaming.runtime.tasks. StreamTask.handleAsyncException(StreamTask.java:1091)     at org.apache.flink.streaming.runtime.tasks. StreamTask.invokeProcessingTimeCallback(StreamTask.java:1222)     at org.apache.flink.streaming.runtime.tasks. StreamTask.lambda$null$16(StreamTask.java:1211)     at org.apache.flink.streaming.runtime.tasks. StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)     at org.apache.flink.streaming.runtime.tasks.mailbox. Mail.run(Mail.java:78)     at org.apache.flink.streaming.runtime.tasks.mailbox. MailboxProcessor.processMail(MailboxProcessor.java:270)     at org.apache.flink.streaming.runtime.tasks.mailbox. MailboxProcessor.runMailboxStep(MailboxProcessor.java:190)     at org.apache.flink.streaming.runtime.tasks.mailbox. MailboxProcessor.runMailboxLoop(MailboxProcessor.java:181)     at org.apache.flink.streaming.runtime.tasks. StreamTask.runMailboxLoop(StreamTask.java:566)     at org.apache.flink.streaming.runtime.tasks. StreamTask.invoke(StreamTask.java:536)     at org.apache.flink.runtime.taskmanager. Task.doRun(Task.java:721)     at org.apache.flink.runtime.taskmanager. Task.run(Task.java:546)     at java.lang.Thread.run(Thread.java:748) Caused by: TimerException{java.lang.NumberFormatException}     ...  12 more Caused by: java.lang.NumberFormatException     at java.math. BigDecimal.<init>(BigDecimal.java:497)     at java.math. BigDecimal.<init>(BigDecimal.java:827)     at scala.math. BigDecimal$.decimal(BigDecimal.scala:50)     at scala.math. BigDecimal$.decimal(BigDecimal.scala:53)     at scala.math. BigDecimal$.double2bigDecimal(BigDecimal.scala:341)     at com.xxx. ProcessPedestrianVisitor.$anonfun$trajectoryReduce$4(PedestrianAnalyzer.scala:198)     at com.xxx. ProcessPedestrianVisitor.$anonfun$trajectoryReduce$4$adapted(PedestrianAnalyzer.scala:198)     at scala.collection. TraversableLike.$anonfun$filterImpl$1(TraversableLike.scala:247)     at scala.collection.immutable. List.foreach(List.scala:388)     at scala.collection. TraversableLike.filterImpl(TraversableLike.scala:246)     at scala.collection. TraversableLike.filterImpl$(TraversableLike.scala:244)     at scala.collection. AbstractTraversable.filterImpl(Traversable.scala:104)     at scala.collection. TraversableLike.filter(TraversableLike.scala:258)     at scala.collection. TraversableLike.filter$(TraversableLike.scala:258)     at scala.collection. AbstractTraversable.filter(Traversable.scala:104)     at com.xxx. ProcessPedestrianVisitor.$anonfun$trajectoryReduce$2(PedestrianAnalyzer.scala:198)     at scala.collection. TraversableLike.$anonfun$map$1(TraversableLike.scala:233)     at scala.collection.immutable. NumericRange.foreach(NumericRange.scala:70)     at scala.collection. TraversableLike.map(TraversableLike.scala:233)     at scala.collection. TraversableLike.map$(TraversableLike.scala:226)     at scala.collection. AbstractTraversable.map(Traversable.scala:104)     at com.xxx. ProcessPedestrianVisitor.$anonfun$trajectoryReduce$1(PedestrianAnalyzer.scala:197)     at scala.collection. TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)     at scala.collection.immutable. NumericRange.foreach(NumericRange.scala:70)     at scala.collection. TraversableLike.flatMap(TraversableLike.scala:240)     at scala.collection. TraversableLike.flatMap$(TraversableLike.scala:237)     at scala.collection. AbstractTraversable.flatMap(Traversable.scala:104)     at com.xxx. ProcessPedestrianVisitor.trajectoryReduce(PedestrianAnalyzer.scala:196)     at com.xxx. ProcessPedestrianVisitor.process(PedestrianAnalyzer.scala:120)     at com.xxx. ProcessPedestrianVisitor.process(PedestrianAnalyzer.scala:98)     at org.apache.flink.streaming.api.scala.function.util. ScalaProcessWindowFunctionWrapper.process(ScalaProcessWindowFunctionWrapper.scala:63)     at org.apache.flink.streaming.runtime.operators.windowing.functions. InternalIterableProcessWindowFunction.process(InternalIterableProcessWindowFunction.java:50)     at org.apache.flink.streaming.runtime.operators.windowing.functions. InternalIterableProcessWindowFunction.process(InternalIterableProcessWindowFunction.java:32)     at org.apache.flink.streaming.runtime.operators.windowing. EvictingWindowOperator.emitWindowContents(EvictingWindowOperator.java:359)     at org.apache.flink.streaming.runtime.operators.windowing. EvictingWindowOperator.onProcessingTime(EvictingWindowOperator.java:318)     at org.apache.flink.streaming.api.operators. InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:260)     at org.apache.flink.streaming.runtime.tasks. StreamTask.invokeProcessingTimeCallback(StreamTask.java:1220)     ...  11 more

The error message is:

 Caused by: java.lang.NumberFormatException

The error location is:

 com.xxx.ProcessPedestrianVisitor.$anonfun$trajectoryReduce$2(PedestrianAnalyzer.scala:198)

That is, in trajectoryReduce In this function, there is an exception of number formatting

Solutions

Two steps, first emergency repair, then code repair.

First, emergency recovery, the process is as follows:
1. Cancel job first
2. Restart a new job
3. Observe whether this situation still exists

The code repair process is as follows:
1. Fix the code and read the abnormal function
2. Determine the location of abnormal points, avoid problems, and finally use if Judge and exclude abnormal data into function processing
3. Test whether the abnormal points are avoided
4. Release, repair and go online

👊 Closure~

This article is written by Chakhsu Lau Creation, adoption Knowledge Sharing Attribution 4.0 International License Agreement.
All articles on this website are original or translated by this website, except for the reprint/source. Please sign your name before reprinting.