multiple-pipeline-capture: Fail to find 4 pipelines still alive, only 3 left

[mengdonglin]

This is a random issue on multiple platforms.

e.g. on EHL RVP and CML-U notebook Olym* in daily test on 2020-10-25, report ID 613.

2020-10-25 23:07:04 UTC [REMOTE_INFO] /home/ubuntu/sof-test/test-case/multiple-pipeline-capture.sh will use topology /lib/firmware/intel/sof-tplg/sof-ehl-rt5660.tplg to run the test case
2020-10-25 23:07:04 UTC [REMOTE_INFO] Pipeline list to ignore is specified, will ignore 'pcm=HDA Digital' in test case
2020-10-25 23:07:04 UTC [REMOTE_INFO] Run command to get pipeline parameters
2020-10-25 23:07:04 UTC [REMOTE_COMMAND] sof-tplgreader.py /lib/firmware/intel/sof-tplg/sof-ehl-rt5660.tplg -f 'type:playback' -b ' pcm:HDA Digital' -s 0 -e
2020-10-25 23:07:04 UTC [REMOTE_INFO] Testing: Headset [hw:0,0]
2020-10-25 23:07:04 UTC [REMOTE_COMMAND] aplay -D hw:0,0 -c 2 -r 48000 -f S16_LE /dev/zero -q
2020-10-25 23:07:04 UTC [REMOTE_INFO] pipeline start sleep 0.5s for device wakeup
2020-10-25 23:07:09 UTC [REMOTE_INFO] checking pipeline status
2020-10-25 23:07:09 UTC [REMOTE_INFO] preparing sleep 5
2020-10-25 23:07:14 UTC [REMOTE_INFO] checking pipeline status again
/home/ubuntu/sof-test/test-case/multiple-pipeline-capture.sh: line 103: 66312 Killed                  "${APP_LST[$1]}" -D $dev -c $channel -r $rate -f $fmt "${DEV_LST[$1]}" -q
/home/ubuntu/sof-test/test-case/multiple-pipeline-capture.sh: line 103: 66321 Killed                  "${APP_LST[$1]}" -D $dev -c $channel -r $rate -f $fmt "${DEV_LST[$1]}" -q
/home/ubuntu/sof-test/test-case/multiple-pipeline-capture.sh: line 103: 66330 Killed                  "${APP_LST[$1]}" -D $dev -c $channel -r $rate -f $fmt "${DEV_LST[$1]}" -q
/home/ubuntu/sof-test/test-case/multiple-pipeline-capture.sh: line 103: 66350 Killed                  "${APP_LST[$1]}" -D $dev -c $channel -r $rate -f $fmt "${DEV_LST[$1]}" -q
2020-10-25 23:07:14 UTC [REMOTE_INFO] ===== Testing: (Loop: 32/50) =====
2020-10-25 23:07:14 UTC [REMOTE_INFO] /home/ubuntu/sof-test/test-case/multiple-pipeline-capture.sh will use topology /lib/firmware/intel/sof-tplg/sof-ehl-rt5660.tplg to run the test case
2020-10-25 23:07:14 UTC [REMOTE_INFO] Pipeline list to ignore is specified, will ignore 'pcm=HDA Digital' in test case
2020-10-25 23:07:14 UTC [REMOTE_INFO] Run command to get pipeline parameters
2020-10-25 23:07:14 UTC [REMOTE_COMMAND] sof-tplgreader.py /lib/firmware/intel/sof-tplg/sof-ehl-rt5660.tplg -f 'type:capture' -b ' pcm:HDA Digital' -s 0 -e
2020-10-25 23:07:14 UTC [REMOTE_INFO] Testing: Headset [hw:0,0]
2020-10-25 23:07:14 UTC [REMOTE_COMMAND] arecord -D hw:0,0 -c 2 -r 48000 -f S16_LE /dev/null -q
2020-10-25 23:07:14 UTC [REMOTE_INFO] Testing: DMIC [hw:0,1]
2020-10-25 23:07:14 UTC [REMOTE_COMMAND] arecord -D hw:0,1 -c 4 -r 48000 -f S16_LE /dev/null -q
2020-10-25 23:07:14 UTC [REMOTE_INFO] Testing: DMIC16kHz [hw:0,2]
2020-10-25 23:07:14 UTC [REMOTE_COMMAND] arecord -D hw:0,2 -c 2 -r 16000 -f S16_LE /dev/null -q
2020-10-25 23:07:14 UTC [REMOTE_INFO] /home/ubuntu/sof-test/test-case/multiple-pipeline-capture.sh will use topology /lib/firmware/intel/sof-tplg/sof-ehl-rt5660.tplg to run the test case
2020-10-25 23:07:14 UTC [REMOTE_INFO] Pipeline list to ignore is specified, will ignore 'pcm=HDA Digital' in test case
2020-10-25 23:07:14 UTC [REMOTE_INFO] Run command to get pipeline parameters
2020-10-25 23:07:14 UTC [REMOTE_COMMAND] sof-tplgreader.py /lib/firmware/intel/sof-tplg/sof-ehl-rt5660.tplg -f 'type:playback' -b ' pcm:HDA Digital' -s 0 -e
2020-10-25 23:07:14 UTC [REMOTE_INFO] Testing: Headset [hw:0,0]
2020-10-25 23:07:14 UTC [REMOTE_COMMAND] aplay -D hw:0,0 -c 2 -r 48000 -f S16_LE /dev/zero -q
2020-10-25 23:07:14 UTC [REMOTE_INFO] pipeline start sleep 0.5s for device wakeup
2020-10-25 23:07:20 UTC [REMOTE_INFO] checking pipeline status
2020-10-25 23:07:20 UTC [REMOTE_INFO] preparing sleep 5
2020-10-25 23:07:25 UTC [REMOTE_ERROR] Target pipeline count: 4, current process count: 3
/home/ubuntu/sof-test/test-case/multiple-pipeline-capture.sh: line 96: 66463 Killed                  "${APP_LST[$1]}" -D $dev -c $channel -r $rate -f $fmt "${DEV_LST[$1]}" -q
2020-10-25 23:07:25 UTC [REMOTE_ERROR] Starting func_exit_handler(), exit status=1, FUNCNAME stack:
/home/ubuntu/sof-test/case-lib/hijack.sh: line 17: 66434 Killed                  "${APP_LST[$1]}" -D $dev -c $channel -r $rate -f $fmt "${DEV_LST[$1]}" -q
/home/ubuntu/sof-test/case-lib/hijack.sh: line 17: 66443 Killed                  "${APP_LST[$1]}" -D $dev -c $channel -r $rate -f $fmt "${DEV_LST[$1]}" -q
/home/ubuntu/sof-test/case-lib/hijack.sh: line 23: 66424 Killed                  "${APP_LST[$1]}" -D $dev -c $channel -r $rate -f $fmt "${DEV_LST[$1]}" -q
2020-10-25 23:07:25 UTC [REMOTE_ERROR]  func_error_exit()  @  /home/ubuntu/sof-test/test-case/multiple-pipeline-capture.sh
2020-10-25 23:07:25 UTC [REMOTE_ERROR]  main()  @  /home/ubuntu/sof-test/test-case/multiple-pipeline-capture.sh:140

[aiChaoSONG]

Try to add some debug log, to see which pipeline is dead.

[plbossart]

I ran 500 iterations on GLK, for which this error was reported in one daily test log. No luck, it's a PASS status. Will try with longer values.

[plbossart]

Of course Murphy's Law applies, failure detected on GLK at the 552th iteration - SOF firmware master and topic/sof-dev.

2020-11-03 03:26:26 UTC [INFO] ===== Testing: (Loop: 552/5000) =====
2020-11-03 03:26:26 UTC [INFO] /root/sof-test/test-case/multiple-pipeline-capture.sh will use topology /lib/firmware/intel/sof-tplg/sof-glk-da7219.tplg to run the test case
2020-11-03 03:26:26 UTC [INFO] Pipeline list to ignore is specified, will ignore 'pcm=HDA Digital' in test case
2020-11-03 03:26:26 UTC [INFO] Run command to get pipeline parameters
2020-11-03 03:26:26 UTC [COMMAND] sof-tplgreader.py /lib/firmware/intel/sof-tplg/sof-glk-da7219.tplg -f 'type:capture' -b ' pcm:HDA Digital' -s 0 -e
2020-11-03 03:26:26 UTC [INFO] Testing: Headset [hw:0,1]
2020-11-03 03:26:26 UTC [COMMAND] arecord -D hw:0,1 -c 2 -r 48000 -f S16_LE /dev/null -q
2020-11-03 03:26:26 UTC [INFO] Testing: DMIC [hw:0,99]
2020-11-03 03:26:26 UTC [COMMAND] arecord -D hw:0,99 -c 4 -r 48000 -f S16_LE /dev/null -q
2020-11-03 03:26:26 UTC [INFO] /root/sof-test/test-case/multiple-pipeline-capture.sh will use topology /lib/firmware/intel/sof-tplg/sof-glk-da7219.tplg to run the test case
2020-11-03 03:26:26 UTC [INFO] Pipeline list to ignore is specified, will ignore 'pcm=HDA Digital' in test case
2020-11-03 03:26:26 UTC [INFO] Run command to get pipeline parameters
2020-11-03 03:26:26 UTC [COMMAND] sof-tplgreader.py /lib/firmware/intel/sof-tplg/sof-glk-da7219.tplg -f 'type:playback' -b ' pcm:HDA Digital' -s 0 -e
2020-11-03 03:26:26 UTC [INFO] Testing: Speakers [hw:0,0]
2020-11-03 03:26:26 UTC [COMMAND] aplay -D hw:0,0 -c 2 -r 48000 -f S16_LE /dev/zero -q
2020-11-03 03:26:26 UTC [INFO] Testing: Headset [hw:0,1]
2020-11-03 03:26:26 UTC [COMMAND] aplay -D hw:0,1 -c 2 -r 48000 -f S16_LE /dev/zero -q
2020-11-03 03:26:26 UTC [INFO] pipeline start sleep 0.5s for device wakeup
2020-11-03 03:26:31 UTC [INFO] checking pipeline status
2020-11-03 03:26:31 UTC [INFO] preparing sleep 5
2020-11-03 03:26:36 UTC [ERROR] Target pipeline count: 4, current process count: 3
/root/sof-test/test-case/multiple-pipeline-capture.sh: line 96: 125680 Killed                  "${APP_LST[$1]}" -D $dev -c $channel -r $rate -f $fmt "${DEV_LST[$1]}" -q
/root/sof-test/test-case/multiple-pipeline-capture.sh: line 96: 125689 Killed                  "${APP_LST[$1]}" -D $dev -c $channel -r $rate -f $fmt "${DEV_LST[$1]}" -q
2020-11-03 03:26:36 UTC [ERROR] Starting func_exit_handler(), exit status=1, FUNCNAME stack:
/root/sof-test/case-lib/hijack.sh: line 23: 125651 Killed                  "${APP_LST[$1]}" -D $dev -c $channel -r $rate -f $fmt "${DEV_LST[$1]}" -q
/root/sof-test/case-lib/hijack.sh: line 23: 125660 Killed                  "${APP_LST[$1]}" -D $dev -c $channel -r $rate -f $fmt "${DEV_LST[$1]}" -q
2020-11-03 03:26:36 UTC [ERROR]  func_error_exit()  @  /root/sof-test/test-case/multiple-pipeline-capture.sh
2020-11-03 03:26:36 UTC [ERROR]  main()  @  /root/sof-test/test-case/multiple-pipeline-capture.sh:140
2020-11-03 03:26:36 UTC [INFO] Starting /usr/local/bin/sof-logger  -l /etc/sof/sof-glk.ldc -o /root/sof-test/logs/multiple-pipeline-capture/2020-11-02-19:47:14-8753/etrace.txt
2020-11-03 03:26:37 UTC [INFO] pkill -TERM sof-logger
Terminated
2020-11-03 03:26:39 UTC [INFO] nlines=7 /root/sof-test/logs/multiple-pipeline-capture/2020-11-02-19:47:14-8753/etrace.txt
2020-11-03 03:26:39 UTC [INFO] Test Result: FAIL!

multiple-pipeline-capture-glk.tar.gz

[plbossart]

And unfortunately there is zero information on what the errors might be in sof-test logs. @aiChaoSONG we really need to update the script to provide more information on which capture failed.

[aiChaoSONG]

@plbossart I have added some debug info in my personal sof-test repo to this case, and also trying to reproduce this issue on some platforms, check report 663, 662, 661, will test more.
Note: CI now supports to use my personal sof-test to run a test plan

[marc-hb]

... to provide more information on which capture failed.

I just submitted two-lines pgrep-a #507 which helps.

And unfortunately there is zero information on what the errors might be in sof-test logs.

I checked and there's no silly /dev/null redirection this time, so this is not a problem with sof-test this time.

Is there any chance --quiet would silence errors too? --quiet options do not normally silence errors but man aplay is very vague.

I have added some debug info in my personal sof-test repo to this case

More specifically?

[fredoh9]

I saw this problem in ICL_HDA_HDA/WHL_RVP_HDA with multiple-pipeline-capture.sh recently. #507 will help debug which process might be the problem but still I don't see any error message.