Skip to content

ipu7_psys_ioctl concurrency race - #99

Open
arunt1 wants to merge 1 commit into
intel:mainfrom
arunt1:cvs
Open

ipu7_psys_ioctl concurrency race #99
arunt1 wants to merge 1 commit into
intel:mainfrom
arunt1:cvs

Conversation

@arunt1

@arunt1 arunt1 commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Kernel panic occurred when launching camera pipline on kernel 7.2.0 version:

Issue
1.A race between ioctl entry and file-handle teardown (release) still exists.
2.The current lifetime guard can still allow a stale file-handle pointer to reach mutex_lock under tight close/ioctl concurrency.
Recommended driver fix:

Fix
1.Use a strict “open reference + in-flight refs” model.
2.In ioctl, acquire ref with refcount_inc_not_zero() before using file-handle state.
3.Re-check closing after ref acquisition; if closing, drop ref and abort.
3. In release, set closing first, drop the open reference, and wait for refcount to reach zero before freeing resources.
4.Ensure every ioctl exit path drops the ref exactly once.

Error Logs
/dev/video32
sudo: preserving the entire environment is not supported, '-E' is ignored
[08-20 15:32:02.947] CamHAL[INF] CameraParserInvoker: getAvailableSensors, Found IPU: IPU7
[08-20 15:32:02.947] CamHAL[INF] CameraParserInvoker: parseSensors: I will Load config file: sensors/ov08x40-uf.json
[08-20 15:32:02.947] CamHAL[INF] CameraSensorsParser: resolveCsiPortAndI2CBus: I2CBus:14-0010 <=> CSI Port:0
[08-20 15:32:02.947] CamHAL[INF] CameraParserInvoker: parseSensors, sensors/ov08x40-uf.json loaded!
Setting pipeline to PAUSED ...
[08-20 15:32:02.960] CamHAL[INF] AiqInitData: aiqb file name ov08x40.aiqb
[08-20 15:32:02.961] CamHAL[INF] AiqInitData: aiqb file name ov08x40.aiqb
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[08-20 15:32:03.7] CamHAL[WAR] GraphUtils: getFourccFmt: no fourcc for resourceId 0, term 25
[08-20 15:32:03.7] CamHAL[WAR] Utils: bpl defaulting to width for format:GREY
[08-20 15:32:03.7] CamHAL[WAR] GraphUtils: getFourccFmt: no fourcc for resourceId 0, term 22
[08-20 15:32:03.7] CamHAL[WAR] Utils: bpl defaulting to width for format:GREY
[ 50.689338] UBSAN: array-index-out-of-bounds in /home/buildnode2/jenkins/workspace/USER_BUILDS/src/kernel/kernel-nvl/kernel/locking/osq_lock.c:41:9
[ 50.702714] index -30225 is out of range for type 'long unsigned int [8192]'
[ 50.709990] Oops: general protection fault, probably for non-canonical address 0x736372ff8ef144a3: 0000 [#1] SMP NOPTI
[ 50.720802] CPU: 6 UID: 0 PID: 4052 Comm: CameraHAL_13097 Kdump: loaded Tainted: G WC OE 7.2.0-camera #2 PREEMPT(lazy)
[ 50.732821] Tainted: [W]=WARN, [C]=CRAP, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[ 50.739936] Hardware name: Intel Corporation Nova Lake Client Platform/NVL UPH/UL LP5X T3 MEMSKT RVP5, BIOS NVLPFWR1.R00.3320.D20.2608040443 08/04/2026
[ 50.753615] RIP: 0010:osq_lock+0x6e/0x1a0
[ 50.757673] Code: c7 c1 40 e3 82 84 49 89 fa 48 98 48 89 e5 48 83 ec 20 48 3d 00 20 00 00 0f 83 09 01 00 00 4c 03 0c c5 e0 f0 c7 83 4d 89 48 08 <4d> 89 01 41 8b 40 10 85 c0 0f 85 a6 00 00 00 65 4c 8b 1d 63 a3 72
[ 50.776605] RSP: 0018:ffffce1a4fc779a0 EFLAGS: 00010207
[ 50.781886] RAX: ffffffffffff89ef RBX: 0000000000000002 RCX: 0000000000000000
[ 50.789094] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 50.796312] RBP: ffffce1a4fc779c0 R08: ffff89f370334340 R09: 736372ff8ef144a3
[ 50.803527] R10: ffff89f00d73380c R11: 0000000000000000 R12: ffff89f00d733800
[ 50.810743] R13: ffff89f00d73380c R14: ffff89f01da7dc00 R15: ffff89f0286030c0
[ 50.817958] FS: 0000771ecd8d76c0(0000) GS:ffff89f3ebb06000(0000) knlGS:0000000000000000
[ 50.826130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 50.831948] CR2: 0000771eac001058 CR3: 0000000106805002 CR4: 0000000108f72ef0
[ 50.839170] PKRU: 55555554
[ 50.841925] Call Trace:
[ 50.844420]
[ 50.846569] __mutex_lock.constprop.0+0xc8/0x920
[ 50.851239] __mutex_lock_slowpath+0x13/0x20

[ 50.855559] mutex_lock+0x3b/0x50
[ 50.858934] ipu7_psys_ioctl+0x6e8/0x11a0 [intel_ipu7_psys]
[ 50.864570] __x64_sys_ioctl+0xa3/0x100
[ 50.868452] x64_sys_call+0x103b/0x2390
[ 50.872333] do_syscall_64+0xdd/0x640
[ 50.876038] ? __mod_memcg_lruvec_state+0x103/0x250
[ 50.880974] ? vm_map_ram+0x16b/0xb30
[ 50.884680] ? lruvec_stat_mod_folio+0xa6/0x100
[ 50.889260] ? folio_add_lru+0x64/0x80
[ 50.893057] ? folio_add_lru_vma+0x73/0xa0
[ 50.897204] ? map_anon_folio_pte_nopf+0xc3/0x170
[ 50.901960] ? do_anonymous_page+0x109/0xa00
[ 50.906292] ? __pte_offset_map+0x1c/0x130
[ 50.910440] ? __handle_mm_fault+0xb9c/0x1010
[ 50.914862] ? count_memcg_events+0xea/0x240
[ 50.919181] ? handle_mm_fault+0x1ec/0x2f0
[ 50.923323] ? do_user_addr_fault+0x2ee/0x830
[ 50.927728] ? irqentry_exit+0x3f/0x7b0
[ 50.931608] ? exc_page_fault+0x94/0x1c0
[ 50.935577] ? do_syscall_64+0x94/0x640
[ 50.939461] fred_entry_from_user+0xa8/0x160
[ 50.943778] asm_fred_entrypoint_user+0x41/0x41

@arunt1 arunt1 changed the title Cvs ipu7_psys_ioctl concurrency race Aug 20, 2026

@avinashiitk avinashiitk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we keep this patch separate.."media: intel: ipu7: psys: harden ioctl argument handling"? PR->99 is for IOCTL and Release lifetime race.

struct ipu7_psys_fh {
struct ipu7_psys *psys;
struct mutex mutex; /* Protects bufmap & kcmds fields */
refcount_t refcnt;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use kref instead of refcount_t? (https://docs.kernel.org/core-api/kref.html)

@avinashiitk avinashiitk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use kref instead of refcount_t? (https://docs.kernel.org/core-api/kref.html)

dseum added a commit to dseum/nixos-hardware that referenced this pull request Aug 26, 2026
Apply the three commits from intel/ipu7-drivers#99 to the pre-ABI-break driver revision. They hold file handles across in-flight ioctls, serialize release with those operations, and validate ioctl arguments without advancing to the incompatible shared-bus layout.
@arunt1

arunt1 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

This issue root caused as struct ipu7_bus_device structure members are not matching exactly , few structure members are missing kernel tree code and external github psys driver.

mutex lock variable got corrupted when calling gstreamer/ psys ioctl.

Following change is fixing the issue

From cf03720bc8ad3ac9ba850334857f59432730f342 Mon Sep 17 00:00:00 2001
From: Arun T arun.t@intel.com
Date: Fri, 28 Aug 2026 21:11:15 +0530
Subject: [PATCH] ipu7: add fw task buffer lock for psys queue access

Add psys task-buffer locking fields to ipu7_bus_device to protect
firmware task queue access and prevent mutex-related crashes during psys operations.

Signed-off-by: Arun T arun.t@intel.com

drivers/staging/media/ipu7/ipu7-bus.h | 2 ++
1 file changed, 2 insertions(+)

diff --git a/drivers/staging/media/ipu7/ipu7-bus.h b/drivers/staging/media/ipu7/ipu7-bus.h
index 45157df16e90..6e03dad54b7e 100644
--- a/drivers/staging/media/ipu7/ipu7-bus.h
+++ b/drivers/staging/media/ipu7/ipu7-bus.h
@@ -46,6 +46,8 @@ struct ipu7_bus_device {
struct ia_gofo_boot_config *boot_config;
dma_addr_t boot_config_dma_addr;
u32 boot_config_size;

  • struct mutex acquire_fw_task_buffer_lock;
  • unsigned int (*get_running_fw_task_count)(struct ipu7_bus_device *adev);
    };
    struct ipu7_auxdrv_data {

This change in kernel tree code , we will push this patch to mlist

mutex_lock(&psys->acquire_fw_task_buffer_lock);
tq = ipu7_psys_get_task_queue(fh->ip, task);
mutex_unlock(&psys->adev->acquire_fw_task_buffer_lock);
mutex_unlock(&psys->acquire_fw_task_buffer_lock);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you run checkpatch.pl script? Looks like extra space or tab here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@manikx , Resolved check patch error and updated with commit ID (52295d3)

Add psys task-buffer locking fields to ipu7_bus_device to protect
firmware task queue access and prevent mutex-related crashes during psys operations.

Signed-off-by: Arun T <arun.t@intel.com>
@manikx

manikx commented Sep 3, 2026

Copy link
Copy Markdown

Approved

@manikx manikx left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No Comments

@hao-yao hao-yao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arunt1 Hi, how about apply this patch to your kernel v7.2?

@arunt1

arunt1 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

@arunt1 Hi, how about apply this patch to your kernel v7.2?

@hao-yao , This patch is solving issue.

Actual issue is acquire_fw_task_buffer_lock and get_running_fw_task_count is not added in struct ipu7_bus_device kernel tree code (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/media/ipu7/ipu7-bus.h?h=v7.3-rc1), so struct ipu7_bus_device fileds are mismatching both kernel tree and github ipu7-drivers.

I already requested below change to push kernel mailing list

Sakari reviewed and provided comments:

This is email chain content.
Hi Arun,

On Fri, Aug 28, 2026 at 12:02:31PM +0300, T, Arun wrote:

Kernel tree :
drivers/staging/media/ipu7/ipu7-bus.h

struct ipu7_bus_device {
struct auxiliary_device auxdev;
const struct auxiliary_driver *auxdrv;
const struct ipu7_auxdrv_data *auxdrv_data;
struct list_head list;
enum ipu7_subsys subsys;
void *pdata;
struct ipu7_mmu *mmu;
struct ipu7_device *isp;
const struct ipu_buttress_ctrl *ctrl;
u64 dma_mask;
struct sg_table fw_sgt;
u32 fw_entry;
struct ipu7_syscom_context *syscom;
struct ia_gofo_boot_config *boot_config;
dma_addr_t boot_config_dma_addr;
u32 boot_config_size;
};

Psys (external github)
./drivers/media/pci/intel/ipu7/ipu7-bus.h
struct ipu7_bus_device {
struct auxiliary_device auxdev;
const struct auxiliary_driver *auxdrv;
const struct ipu7_auxdrv_data *auxdrv_data;
struct list_head list;
enum ipu7_subsys subsys;
void *pdata;
struct ipu7_mmu *mmu;
struct ipu7_device *isp;
const struct ipu_buttress_ctrl *ctrl;
u64 dma_mask;
struct sg_table fw_sgt;
u32 fw_entry;
struct ipu7_syscom_context *syscom;
struct ia_gofo_boot_config *boot_config;
dma_addr_t boot_config_dma_addr;
u32 boot_config_size;

/* Serialize FW message buffer or task queue acquisition against

  • TLB invalidation.
    */
    struct mutex acquire_fw_task_buffer_lock;
    unsigned int (*get_running_fw_task_count)(struct ipu7_bus_device
    *adev); };

I believe you'll need to carry this patch in a downstream kernel, it has no use in upstream.

--
Kind regards,

Sakari Ailus

--- a/drivers/staging/media/ipu7/ipu7-bus.h
+++ b/drivers/staging/media/ipu7/ipu7-bus.h
@@ -46,6 +46,12 @@ struct ipu7_bus_device {
struct ia_gofo_boot_config *boot_config;
dma_addr_t boot_config_dma_addr;
u32 boot_config_size;
+

  • /* Serialize FW message buffer or task queue acquisition against
    • TLB invalidation.
  • */
  • struct mutex acquire_fw_task_buffer_lock;
  • unsigned int (*get_running_fw_task_count)(struct ipu7_bus_device *adev);
    };

@hao-yao i think Better we can use this commit (52295d3) to ipu7-driver code, this patch solve the issue.

@hao-yao

hao-yao commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

@arunt1 Patches in v7.0.0_iot was to solve IPU7&8 multiple streams start/stop issues. Besides, patch 52295d3 is not fully implemented, at least mutex init was missing.

@avinashiitk avinashiitk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to wait until a consensus is reached among all reviewers

@avinashiitk avinashiitk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to wait until a consensus is reached among reviewers.

@avinashiitk avinashiitk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arunt1 is it reproducible only with "Synaptic Sabre"? pure CSI port and camera AIC does not cause this kernel panic?

@arunt1

arunt1 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

@arunt1 is it reproducible only with "Synaptic Sabre"? pure CSI port and camera AIC does not cause this kernel panic?

@avinashiitk , currently it is reproducible only on synaptic Sabre card, and more time related.
It may come on MIPI also, acquire_fw_task_buffer_lock struct field is not properly mapped to kernel tree and ipu7-drivers code

@arunt1 arunt1 closed this Sep 4, 2026
@arunt1 arunt1 reopened this Sep 4, 2026
@arunt1

arunt1 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

@saragsapre we are close to NVL PV stage, we cannot hold any patches as Internal , it take long time to go for upstream , I think better we can consider this patch as workaround, we can revert this patch once acquire_fw_task_buffer_lock varaible got up streamed to mainline

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants