PHP analyzes Nginx logs Author: Chuwen Time: 2022-01-18 Classification: PHP comment Source address: https://gist.github.com/zwjzxh520/4444e276db0db5423dfc3dd0e437408d <script src=" https://gist.github.com/zwjzxh520/4444e276db0db5423dfc3dd0e437408d.js "></script> ```php <? php /** *Nginx log file analysis. *Analyze the corresponding log content according to the configured nginx log format. Each variable */ class NginxLog { protected static $br = "\n"; /** *Analyze the nginx access log according to the log format. *Format example: "$remote_addr" $request_time - $remote_user d [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" *@ param array $log Access log array *@ param string $format Access log format *@ return array Access log array after analysis. Key is the name of the access log variable, and value is the log information corresponding to the variable */ public function analysisAccess($log, $format) { $logVars = $this->parseFormat($format); $log = explode(self::$br, $log); $formatLen = strlen($format); $logArr = []; foreach ($log as $lineNum=>$line) { $line = trim($line); if (empty($line)) { continue; } //Log content offset $linePos = 0; //Log format offset $formatPos = 0; foreach ($logVars as $var => $info) { //The starting position of the variable content. $linePos += $info['prevLen']; //End position of the log content of this variable if ($info['endChar']) { $varLogEnd = strpos($line, $info['endChar'], $linePos); } else { $varLogEnd = 0; } $logArr[$lineNum][ $var ] = substr($line, $linePos, $varLogEnd ? $varLogEnd - $linePos : null); //Start position of the next variable if ($varLogEnd) { $linePos += $varLogEnd - $linePos; } } } return $logArr; } /** *Analyze nginx error logs. *Because some error messages are divided into multiple lines (such as the stack error of php fatal error), by default, *Use the $mergeLog variable to retain error logs, and combine these error logs into one line for analysis. *If the error log is incomplete, it will not be analyzed. *If you get a complete error log, $mergeLog will be reset. *Provide the $reset parameter to clear the $mergeLog variable. *@ param array $logLines Error log array *@ param boolean $reset Whether to reset the last analysis and retain incomplete error logs *@ return array Error log array after analysis */ public function analysisError($logLines, $reset = false) { $result = []; $yinhaoCount = 2; static $mergeLog = ''; if ($reset) { $mergeLog = ''; } foreach ($logLines as $lineNum => $log) { if (substr_count($log, '"') >= $yinhaoCount) { if ($mergeLog) { $log = $mergeLog.$ log; $mergeLog = ''; } $logResult = $this->parseLineError($log); if (! empty($logResult)) { $result[] = $logResult; } } else { $mergeLog .= $ log."\n"; } } return $result; } /** *Analyze a line of error logs. *The complete log will start with the time. If not, it will not be analyzed. *Return results: * [ *'datetime '=>' Time stamp of error ' *'type '=>' nginx error type ' *'fd'=>'nginx thread id' *'msg '=>' nginx error type description ' *'desc '=>' nginx error detailed prompt ' *'http'=>['http request related information '] *'phperr'=>'php error type, empty if it is not a php error' * ] *@ param string $log a complete error log * @return array [description] */ protected function parseLineError($log) { $datetimeLen = 19; // Date string length $metaPos = $datetimeLen + 1; // Start position of meta character truncation $datetime = substr($log, 0, 19); $result = []; if ($this->isDateTimeFormat($datetime)) { //Date Time $result['datetime'] = strtotime($datetime); $metaEndPos = strpos($log, '"', $metaPos); $errorDescEndPos = strpos($log, '"', $metaEndPos + 1); $metaStr = trim(substr($log, $metaPos, $metaEndPos - $metaPos)); $metaArr = explode(' ', $metaStr); //Error type $result['type'] = substr($metaArr[0], 1, -1); //File descriptor. Nginx process dependency $result['fd'] = $metaArr[1].' '.$ metaArr[2]; //Error Type Description $result['msg'] = rtrim(implode(' ', array_slice($metaArr, 3)), ':'); //Error prompt description $result['desc'] = substr($log, $metaEndPos +1, $errorDescEndPos - $metaEndPos - 1); //Http related information $result['http'] = $this->parseErrorHTTP(trim(substr($log, $errorDescEndPos + 1))); //PHP error, if it is a PHP error $result['phperr'] = $this->getPHPErrorType($result['desc']); } return $result; } /** *Analyze the php error type, and return null if it is not a php error * @param string $str * @return string */ protected function getPHPErrorType($str) { //PHP message: $type = ''; if ('PHP message:' === substr($str, 0, 12)) { //Why is it 17? Because PHP message: PHP Waring, the length of 'PHP' should be added $type = substr($str, 17, strpos($str, ':', 12) - 17); } return $type; } /** *Parse the relevant information of the http request. Generally, from the client: string to the end of the log *The returned results (the following fields may or may not exist, depending on the log): * [ *'0'=>'A prompt message before the client keyword after the nginx error detailed prompt (desc)' *'client'=>'client ip' *'server '=>' Domain name ' *'method '=>' Request method ' *'uri'=>'Requested uri' *'httpver'=>'http version' *'upstream'=>'php fpm address' *'host'=>'host header' *'referer '=>' Reference url address' * ] * @param string $http * @return array */ protected function parseErrorHTTP($http) { //client: $split = strpos($http, 'client:'); $result[0] = substr($http, 0, $split - 2); $other = explode(',', substr($http, $split)); foreach ($other as $value) { $pos = strpos($value, ': '); $key = trim(substr($value, 0, $pos)); $val = trim(substr($value, $pos + 2), '"'); if ('request' === $key) { $result = array_merge($result, $this->parseRequest($val)); } else { $result[$key] = $val; } } return $result; } /** *Parse http request information. It is the resolution of "GET http://www.baidu. com HTTP/1.1". *Return results: * [ *'method '=>' Request method ' *'uri'=>'Requested uri' *'httpver'=>'http version' * ] *@ param string $request Example: "GET http:/www.baidu. com HTTP/1.1" * @return array */ protected function parseRequest($request) { $result = []; $split = ' '; $firstPos = strpos($request, $split); $lastPos = strrpos($request, $split); return [ 'method' => substr($request, 0, $firstPos), 'uri' => trim(substr($request, $firstPos, $lastPos - $firstPos)), 'httpver' => trim(substr($request, $lastPos)), ]; } /** *Check whether the string is in date format. Fixed format: yyyy/mm/dd hh: mm: ss *@ param string $str String to be checked * @return boolean */ protected function isDateTimeFormat($str) { //2016/09/06 00:32:19 $number = '0123456789'; $format = '0000/00/00 00:00:00'; $strlen = strlen($str); $result = true; for($i=0; $ i<$strlen; $i++) { if ($format{$i} === '0') { if (false === strpos($number, $str{$i})) { $result = false; break; } } elseif ($format{$i} !== $str{$i}) { $result = false; break; } } return $result; } /** *Analyze the access log format, get the variable name, and save the starting position of the variable in the log content. *@ param string $format Log format in nginx configuration file * @return array */ public function parseFormat($format) { $varCharList = 'abcdefghijklmnopqrstuvwxyz_$ABCDEFGHIJKLMNOPQRSTUVWXYZ'; $return = []; $formatLen = strlen($format); $logVar = ''; $prevStrLen = 0; for ($i=0; $ i < $formatLen; $i++) { $char = $format{$i}; if (strpos($varCharList, $char) === false || (empty($logVar) && $char != '$')) { if ($logVar) { $return[$logVar]['endChar'] = $char; // The character next to the variable. The position of the character in the log content indicates the end position of the variable $return[$logVar]['prevLen'] = $prevStrLen; // Length of characters ignored before variable $prevStrLen = 1; $logVar = ''; } else { $prevStrLen++; } } else { $logVar .= $ char; } } if (! empty($logVar)) { $return[$logVar]['endChar'] = ''; $return[$logVar]['prevLen'] = $prevStrLen; $logVar = ''; $prevStrLen = 0; } return $return; } /** *Read large files. *Set only one of the $pos parameter and $startLine parameter, otherwise the result may not be as expected. *When $pos is specified, to ensure that a complete row is read, all data in the row where the position is located will be read. *@ param string $filepath File absolute path * @param integer $pos Start position of reading *@ param integer $startLine Start line read * @return [type] [description] */ public function readBigFile($filepath, $pos = 0, $startLine = 1) { $handle = fopen($filepath, 'rb'); $br = self::$br; $brLen = strlen($br); $lastShortLine = ''; $blockLen = 20480; // Size of data block read each time if ($handle) { if ($pos > 0) { //Move to an appropriate location to ensure that a complete line of data is read $movePos = 0; while($pos - $movePos >= 0) { fseek($handle, $pos - $movePos); $c = fread($handle, 1); if ($c === "\n" || $c === "\r") { break; } $movePos ++; } if ($c !== "\n" && $c !== "\r") { fseek($handle, -1, SEEK_CUR); } unset($movePos); } while ( ($data = fread($handle, $blockLen)) ) { if (! feof($handle)) { $dataSize = strlen($data); //Supplement the incomplete row data read last time if ($lastShortLine){ $data = $lastShortLine.$ data; $dataSize = strlen($data); } $lastBrPos = strrpos($data, $br); if ($lastBrPos !== false && $lastBrPos != $dataSize+$brLen) { $lastShortLine = substr($data, $lastBrPos+$brLen); } else { $lastShortLine = ''; } $result = $lastBrPos !== false ? substr($data, 0, $lastBrPos) : $data; //Locate Start Line if ($startLine > 1) { $rows = substr_count($result, $br); $startLine = $startLine - $rows; if ($startLine > 1) { continue; } else { $startpos = 0; $startLine = abs($startLine); while ($rows - $startLine > 1) { $startpos += strpos($result, $br, $startpos) + $brLen; $startLine++; } $result = substr($result, $startpos); unset($rows, $startLine, $startpos); } } } else { $result = $lastShortLine.$ data; } yield $result; } fclose($handle); } } } $format = '"$remote_addr" $request_time - $remote_user d [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for"'; $log = <<<LOG "192.168.110.1" 0.040 - - d [05/Aug/2016:08:22:54 +0800] "GET /uc_server/data/avatar/006/40/73/36_real_avatar_small.jpg HTTP/1.0" 404 6664 "-" "-" "-" "36.149.107.72" 3.016 - - d [05/Aug/2016:08:22:54 +0800] "GET /cardniu/api/api_splashinterface.php?udid=deviceId-866968026335845-generate-cardniu&systemName=android+OS&systemVersion=4.4.4&productName=Cardniu&productVersion=4&position=KNSQTZXQB&positionList=1&chanelSys=shequ HTTP/1.1" 200 5 " http://www.baidu.com/cardniu/detail.php?tid=552221&utm_source=552221&utm_medium=ribao&utm_campaign=xiaoxi " "Mozilla/5.0 (Linux; Android 4.4.4; A31 Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 feideeAndroid-V4 MymoneySms7.3.2-oppo" "-" "192.168.110.1" 0.040 - - d [05/Aug/2016:08:22:54 +0800] "GET //uc_server/data/avatar/005/86/04/56_real_avatar_small.jpg HTTP/1.0" 404 6664 "-" "-" "-" LOG; // $log = file_get_contents('test.txt'); //Test Access Log $startTime = microtime(true); // var_export( (new NginxLog) -> parseFormat($format) ); var_export( (new NginxLog) -> analysisAccess($log, $format) ); echo microtime(true) - $startTime; //Test Error Log // $logFile = __DIR__.'/../ testdata/nginx_access.log'; $errorLogFile = __DIR__.'/../ testdata/www.baidu.com_error_2016-09-05.log'; $value = <<<LOG 2016/09/05 14:52:52 [error] 17890#0: *1424957037 FastCGI sent in stderr: "PHP message: PHP Fatal error: Uncaught exception 'DbException' with message 'Duplicate entry '240319164' for key 'sid'' in /var/www/html/bbs/source/class/db/db_driver_mysql.php:218 Stack trace: #0 /var/www/html/bbs/source/class/db/db_driver_mysql.php(151): db_driver_mysql->halt('Duplicate entry...', 1062, 'UPDATE pre_com...') #1 /var/www/html/bbs/source/class/db/db_driver_mysql_slave.php(62): db_driver_mysql->query('UPDATE pre_com...', false, false) #2 /var/www/html/bbs/source/class/discuz/discuz_database.php(179): db_driver_mysql_slave->query('UPDATE pre_com...', false, false) #3 /var/www/html/bbs/source/class/discuz/discuz_database.php(102): discuz_database::query('UPDATE pre_com...', '') #4 /var/www/html/bbs/source/class/discuz/discuz_table.php(52): discuz_database::update('common_member', Array, '`uid`='10929277...', false, false) #5 /var/www/html/bbs/m/register_by_email.php(72): discuz_table->update(10929277, Array) #6 /var/www/html/bbs/m/function/ssj_function.php(788): register_by_ssjuser('', '" while reading response header from upstream, client: 192.168.241.97, server: www.baidu.com, request: "POST /m/api/credit.php HTTP/1.1", upstream: " fastcgi://127.0.0.1:9000 ", host: "www.baidu.com" 2016/09/06 05:11:54 [error] 10527#0: *915961403 open() "/var/www/html/cardniu/thread-544114" failed (2: No such file or directory), client: 180.153.205.253, server: www.baidu.com, request: "GET /thread-544114? 10000skip=true-1-1.html HTTP/1.1", host: "www.baidu.com", referrer: " http://www.baidu.com/cardniu/detail.php?tid=544114?10000skip=true " 2016/09/06 06:22:15 [error] 10535#0: *916177977 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 111.127.121.50, server: www.baidu.com, request: "GET /cardniu/api/home.php? mod=spacecp&ac=usergroup&do=expiry HTTP/1.1", upstream: " fastcgi://127.0.0.1:9000 ", host: "www.baidu.com", referrer: " http://www.baidu.com/cardniu/kn_daily.php?stamptime=2016 -09-05&udid=deviceId-868979027115530-generate-cardniu&bankcode=SPD,CMB,CEB" LOG; $startTime = microtime(true); $nginxLog = new NginxLog; $logNum = 0; $phperrs = []; foreach ($nginxLog -> readBigFile($errorLogFile) as $value) { $logString = trim($value); $logArr = explode("\n", $logString); $logNum += count($logArr); foreach ($nginxLog->analysisError($logArr) as $log) { if (! empty($log['phperr'])) { $phperrs[] = $log; } } } echo 'finish. cost time:'.(microtime(true) - $startTime). "\n"; Echo 'Number of logs:'$ logNum."\n"; Echo 'Number of php error logs:'. count ($phperrs) "\n"; var_export($phperrs); ```
Mbox+ReactECharts does not update data Author: Chuwen Time: 2022-01-12 Classification: React comment Tsx is as follows: ``` <ReactECharts option={this.props.shipmentsLookupsChartStore?.getOptions}/> ``` The store is as follows: >If you follow the following 'setOptionsData', there will be a problem. The data has been given, but the view will not be updated ```ts import {action, computed, makeObservable, observable} from "mobx"; export default class ChartStore implements IChart { @observable public options = ChartOption; constructor() { makeObservable(this) } @computed get getOptions(): any { return this.options } @action.bound setOptionsData(data: ChartOptionsData): void { //Set x-axis data if (this.options?. xAxis[0]?. data) { this.options.xAxis[0].data = data.xAxisData; } } } ``` ###Solution >The reason is that the reason for deep and shallow cloning is only for record, and it does not guarantee that what I said is correct First, install 'lodash', and change the 'setOptionsData' method to the following, which will run normally: ```ts @action.bound setOptionsData(data: ChartOptionsData): void { //Deep clone data const newOption = cloneDeep(this.options) //Set x-axis data if (this.options?. xAxis[0]?. data) { newOption.xAxis[0].data = data.xAxisData } this.options = newOption } ```
React+Vite+TypeScript reads environment variables and reports an error: Uncaught ReferenceError: process is not defined Author: Chuwen Time: 2022-01-05 Classification: React comment Uncaught ReferenceError: process is not defined: Relevant sources: https://github.com/vitejs/vite/issues/1973#issuecomment -787571499 Solution: In 'vite. config. json': ```json import {defineConfig, loadEnv} from 'vite' import react from '@vitejs/plugin-react' export default ({mode}: { mode: string }) => { return defineConfig({ define:{ 'process.env':{...process.env, ...loadEnv(mode, process.cwd())} } }); } ``` Then name your environment variable file with 'VITE_', and then use 'process. env VITE_XXX ` Just get it