From 36f65dac27d7274a874567f31322693161c33447 Mon Sep 17 00:00:00 2001 From: DoodleBear Date: Tue, 3 Nov 2026 16:54:26 +0900 Subject: [PATCH 1/3] feat: prepare quoted Lark voice messages for Channel playback Refs #1023 Co-Authored-By: Claude Opus 5.5 (1M context) --- changes/1023-lark-channel-audio.md | 11 ++ docs/client-bridge.md | 2 +- docs/lark-connection.md | 4 + docs/lark-connection.zh.md | 4 + docs/product-im-installation.md | 2 +- packages/core/package.json | 1 + packages/core/src/attachments/lark-audio.ts | 106 ++++++++++++++ packages/core/src/attachments/voice-audio.ts | 15 +- packages/core/src/messaging/channel-target.ts | 4 +- packages/core/src/messaging/dsh-im.ts | 42 +++++- packages/core/src/messaging/outbound.ts | 15 +- .../core/test/channel-bridge-media.test.ts | 129 ++++++++++++++++-- packages/core/test/lark-audio.test.ts | 45 ++++++ packages/core/test/ogg-opus.ts | 47 +++++++ pnpm-lock.yaml | 28 ++++ scripts/dev-im-provider.mjs | 6 +- scripts/product-artifacts.mjs | 6 +- scripts/test/product-artifacts.test.mjs | 14 +- tsdown.config.ts | 3 +- 19 files changed, 442 insertions(+), 42 deletions(-) create mode 100644 changes/1023-lark-channel-audio.md create mode 100644 packages/core/src/attachments/lark-audio.ts create mode 100644 packages/core/test/lark-audio.test.ts create mode 100644 packages/core/test/ogg-opus.ts diff --git a/changes/1023-lark-channel-audio.md b/changes/1023-lark-channel-audio.md new file mode 100644 index 0000000000..77d11a921b --- /dev/null +++ b/changes/1023-lark-channel-audio.md @@ -0,0 +1,11 @@ +--- +section: Added +--- + +## en + +- Prepare and play a Lark voice message quoted by a Bot-mentioned request in its Channel bubble: the Host decodes the Opus original to WAV only on explicit request, shows the platform duration and keeps the original downloadable ([#1023](https://github.com/BotHarness/DeepSeekBot/issues/1023)). + +## zh + +- 在 Channel 请求气泡内准备并播放被 @ Bot 请求引用的 Lark 语音:Host 仅在明确操作后将 Opus 原件解码为 WAV,显示平台时长,原件仍可下载([#1023](https://github.com/BotHarness/DeepSeekBot/issues/1023))。 diff --git a/docs/client-bridge.md b/docs/client-bridge.md index 8eec99185e..4489c2bb41 100644 --- a/docs/client-bridge.md +++ b/docs/client-bridge.md @@ -144,7 +144,7 @@ resolver cannot substitute workspace-linked Core/Client for the tarballs. node scripts/dev-instance.mjs --home /tmp/bh-im-qa --port 31968 --im-provider --json ``` -仅对已停止的隔离 Profile 执行;重启同一 Profile 时仍传入 `--im-provider`,以重复校验。当前 Provider 来源记录为 Git 完整提交 `e655636c061fa0aed8c77c23f7e87cb5abf79b65`,开发资格记录为 DSH `0.2.1-rc.2`,这些来源记录不限制实际启动的 DSH 版本;校验 Bundle、入口及运行时代码 digest 后才启动 Host;不依赖另一份 provider 本地源码。安装摘要明确标注 `upstreamReleased: false`。此完整 fork 保留附件、回显与独立回复身份契约;群/话题上下文读取绑定当前独占 Consumer 生命周期,释放或替换后拒绝返回正在读取的结果;上游 PR 是否合并不阻挡此固定提交使用([#789](https://github.com/BotHarness/BotHarness/issues/789))。默认启动不安装它,原 npm `4.32.0` 仍不满足账号校验和条件发送契约。此入口不会发布包,也不是生产启用许可([ADR-0104](adr/0104-isolated-im-profiles-pin-a-qualified-temporary-provider-fork.md))。 +仅对已停止的隔离 Profile 执行;重启同一 Profile 时仍传入 `--im-provider`,以重复校验。当前 Provider 来源记录为 Git 完整提交 `bb0c7956896f5f5845950a0707837cd7938a11b3`,开发资格记录为 DSH `0.2.1-rc.2`,这些来源记录不限制实际启动的 DSH 版本;校验 Bundle、入口及运行时代码 digest 后才启动 Host;不依赖另一份 provider 本地源码。安装摘要明确标注 `upstreamReleased: false`。此完整 fork 保留附件、回显与独立回复身份契约;群/话题上下文读取绑定当前独占 Consumer 生命周期,释放或替换后拒绝返回正在读取的结果;上游 PR 是否合并不阻挡此固定提交使用([#789](https://github.com/BotHarness/BotHarness/issues/789))。默认启动不安装它,原 npm `4.32.0` 仍不满足账号校验和条件发送契约。此入口不会发布包,也不是生产启用许可([ADR-0104](adr/0104-isolated-im-profiles-pin-a-qualified-temporary-provider-fork.md))。 在 dsh-im 原设置中配置测试应用,并保存、测试仅含测试者与 Bot 的目标群;凭据交由 DSH credentials service,不复制到 BotHarness。打开 PersonaBot Profile → IM connection,选择已认证账号与已测试目标,显式授权后发送一条唯一测试文本。预期 Recent sends 显示 Platform accepted,并在目标群核对同一文本;这不代表送达或已读。停止本次启动摘要中的确切 PID,再以相同参数启动:绑定与发送历史应保留,不自动重发。 diff --git a/docs/lark-connection.md b/docs/lark-connection.md index a1398d35a3..2b6d4c1b95 100644 --- a/docs/lark-connection.md +++ b/docs/lark-connection.md @@ -507,3 +507,7 @@ Preparation acquires up to 25 MiB under current source authorization. Playback t This is a draft video tracer. Native intake, decoded playback, installed-product E2E, matched-theme captures and Human feedback remain pending in [the verification checkpoint](dev/verification/lark-1023-channel-video.md). Audio, direct media messages, runtime transcoding, generated covers and model transcription are not delivered by this slice. The mobile app currently offers the original as a file action. Opening the Bot Inbox's source details does not fetch this video or its cover. That inspector retains explicit original-download actions; prepare playback from the Channel bubble. + +## Referenced voice message in Channel history + +A Lark voice message quoted by a text request that really mentions the Bot appears in the Channel bubble as an audio card with **Prepare playback** and an original download. Preparing fetches the voice (up to 1 MiB) under the same source authorization and converts its Opus audio to WAV on the Host; playback then starts only when you press play. Lark gives no transcript, so the bubble marks it as unavailable; a known length shows as the platform duration. Formats other than Ogg Opus are refused with the original still downloadable. Stop, unbind and revocation follow the file rules above. The Bot Inbox's source details also prepare audio only on an explicit action. diff --git a/docs/lark-connection.zh.md b/docs/lark-connection.zh.md index 5692c0313e..af0f7931d4 100644 --- a/docs/lark-connection.zh.md +++ b/docs/lark-connection.zh.md @@ -472,3 +472,7 @@ Human 查看图片不会让 Bot 获得图片理解能力,也不改变其注意 本片仍是视频草稿候选。真实入站、实际解码播放、安装包 E2E、明暗截图和 Human 反馈待[验收检查点](dev/verification/lark-1023-channel-video.md)补齐。音频、直接媒体消息、运行时转码、生成封面和模型转写未在本片交付;移动端目前提供原件文件操作。 打开 Bot Inbox 的来源详情不会取回此视频或封面;详情中保留明确的原件下载操作,播放请从 Channel 气泡准备。 + +## Channel 历史中的引用语音 + +被真正 @ Bot 的文字请求引用的 Lark 语音,会在 Channel 气泡中显示为音频卡片,提供「准备播放」和原件下载。准备操作在同一来源授权下取得语音(不超过 1 MiB),由 Host 将 Opus 音频转为 WAV;之后需手动点击播放。飞书不提供转写,气泡会标明转写不可用;已知时长显示为平台时长。非 Ogg Opus 格式会被拒绝,原件仍可下载。暂停、解绑和撤销遵守上方文件规则。Bot Inbox 来源详情同样只在明确操作后准备音频。 diff --git a/docs/product-im-installation.md b/docs/product-im-installation.md index 2f8ac5997a..e1bcceb8ea 100644 --- a/docs/product-im-installation.md +++ b/docs/product-im-installation.md @@ -35,7 +35,7 @@ fails before packing. ```bash git clone https://github.com/DoodleBears/dsh-im.git /tmp/bh-im-source -git -C /tmp/bh-im-source checkout e655636c061fa0aed8c77c23f7e87cb5abf79b65 +git -C /tmp/bh-im-source checkout bb0c7956896f5f5845950a0707837cd7938a11b3 npm ci --prefix /tmp/bh-im-source --ignore-scripts --no-audit --no-fund pnpm install --frozen-lockfile pnpm build diff --git a/packages/core/package.json b/packages/core/package.json index f21887e3cf..ddffcfa3f1 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -23,6 +23,7 @@ "@noble/hashes": "2.4.0", "@resvg/resvg-js": "2.6.2", "css-tree": "3.2.1", + "opus-decoder": "0.7.12", "saxes": "6.0.0", "silk-wasm": "3.7.1", "uqr": "0.1.3", diff --git a/packages/core/src/attachments/lark-audio.ts b/packages/core/src/attachments/lark-audio.ts new file mode 100644 index 0000000000..a3dfa0633f --- /dev/null +++ b/packages/core/src/attachments/lark-audio.ts @@ -0,0 +1,106 @@ +import { MessagingError } from '../messaging/provider.js'; +import { decodeInWorker, MAX_VOICE_INPUT_BYTES } from './voice-audio.js'; + +const MAX_PACKETS = 20000; + +interface OggOpus { + channels: number; + preSkip: number; + packets: Uint8Array[]; +} + +export function oggOpusPackets(bytes: Uint8Array): OggOpus { + const ascii = (offset: number, length: number) => + new TextDecoder().decode(bytes.subarray(offset, offset + length)); + if (bytes.length < 28 || ascii(0, 4) !== 'OggS') + throw new MessagingError('audio-codec-unsupported'); + const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); + const packets: Uint8Array[] = []; + let pending: Uint8Array[] = []; + let serial: number | undefined; + let offset = 0; + while (offset < bytes.length) { + if (offset + 27 > bytes.length || ascii(offset, 4) !== 'OggS' || bytes[offset + 4] !== 0) + throw new MessagingError('audio-decode-failed'); + const pageSerial = view.getUint32(offset + 14, true); + if (serial !== undefined && pageSerial !== serial) + throw new MessagingError('audio-codec-unsupported'); + serial = pageSerial; + const segments = bytes[offset + 26]!; + let data = offset + 27 + segments; + if (data > bytes.length) throw new MessagingError('audio-decode-failed'); + for (let index = 0; index < segments; index++) { + const size = bytes[offset + 27 + index]!; + if (data + size > bytes.length) throw new MessagingError('audio-decode-failed'); + pending.push(bytes.subarray(data, data + size)); + data += size; + if (size < 255) { + const packet = new Uint8Array(pending.reduce((total, part) => total + part.length, 0)); + let at = 0; + for (const part of pending) { + packet.set(part, at); + at += part.length; + } + pending = []; + if (packet.length) packets.push(packet); + if (packets.length > MAX_PACKETS + 2) throw new MessagingError('audio-too-large'); + } + } + offset = data; + } + const head = packets[0]; + if ( + !head || + head.length < 19 || + new TextDecoder().decode(head.subarray(0, 8)) !== 'OpusHead' || + head[18] !== 0 || + (head[9] !== 1 && head[9] !== 2) + ) + throw new MessagingError('audio-codec-unsupported'); + const tags = packets[1]; + if (!tags || new TextDecoder().decode(tags.subarray(0, 8)) !== 'OpusTags' || packets.length < 3) + throw new MessagingError('audio-decode-failed'); + return { + channels: head[9]!, + preSkip: head[10]! | (head[11]! << 8), + packets: packets.slice(2), + }; +} + +const DECODE_WORKER = ` +const { parentPort, workerData } = require('node:worker_threads'); +(async () => { + try { + const { OpusDecoder } = await import(workerData.moduleUrl); + const decoder = new OpusDecoder({ + channels: workerData.channels, preSkip: workerData.preSkip, sampleRate: 24000, + }); + await decoder.ready; + const result = decoder.decodeFrames(workerData.packets); + decoder.free(); + const frames = result.samplesDecoded; + if (result.errors.length || !frames || frames * 2 > workerData.maxBytes) + throw new Error('bounded output'); + const pcm = new Uint8Array(frames * 2); + const view = new DataView(pcm.buffer); + for (let frame = 0; frame < frames; frame++) { + let sample = 0; + for (const channel of result.channelData) sample += channel[frame]; + sample = Math.max(-1, Math.min(1, sample / result.channelData.length)); + view.setInt16(frame * 2, Math.round(sample * 32767), true); + } + parentPort.postMessage({ pcm }); + } catch { parentPort.postMessage({ failed: true }); } +})();`; + +export async function decodeLarkVoice(bytes: Uint8Array, signal: AbortSignal): Promise { + signal.throwIfAborted(); + if (!bytes.length || bytes.length > MAX_VOICE_INPUT_BYTES) + throw new MessagingError('audio-too-large'); + const { channels, preSkip, packets } = oggOpusPackets(bytes); + return decodeInWorker( + DECODE_WORKER, + { module: 'opus-decoder', channels, preSkip, packets }, + signal, + ); +} diff --git a/packages/core/src/attachments/voice-audio.ts b/packages/core/src/attachments/voice-audio.ts index 911f42a2ec..008ebf2940 100644 --- a/packages/core/src/attachments/voice-audio.ts +++ b/packages/core/src/attachments/voice-audio.ts @@ -66,11 +66,20 @@ export async function decodeSilkVoice(bytes: Uint8Array, signal: AbortSignal): P if (!bytes.length || bytes.length > MAX_VOICE_INPUT_BYTES) throw new MessagingError('audio-too-large'); silkFrames(bytes); - const moduleUrl = pathToFileURL(createRequire(import.meta.url).resolve('silk-wasm')).href; + return decodeInWorker(DECODE_WORKER, { module: 'silk-wasm', bytes }, signal); +} + +export function decodeInWorker( + source: string, + data: { module: string } & Record, + signal: AbortSignal, +): Promise { + const { module, ...rest } = data; + const moduleUrl = pathToFileURL(createRequire(import.meta.url).resolve(module)).href; return new Promise((resolve, reject) => { - const worker = new Worker(DECODE_WORKER, { + const worker = new Worker(source, { eval: true, - workerData: { moduleUrl, bytes, maxBytes: MAX_VOICE_PCM_BYTES }, + workerData: { ...rest, moduleUrl, maxBytes: MAX_VOICE_PCM_BYTES }, resourceLimits: { maxOldGenerationSizeMb: 64, maxYoungGenerationSizeMb: 16 }, }); let settled = false; diff --git a/packages/core/src/messaging/channel-target.ts b/packages/core/src/messaging/channel-target.ts index 772be120c3..d0ca2e4b8f 100644 --- a/packages/core/src/messaging/channel-target.ts +++ b/packages/core/src/messaging/channel-target.ts @@ -74,6 +74,8 @@ export function bridgeAttachmentKind( if (attachment.mediaType === undefined) return 'file'; if (source.event.video?.itemId === attachment.id && attachment.mediaType === 'video/unknown') return 'video'; + if (source.event.voice?.itemId === attachment.id && attachment.mediaType === 'audio/unknown') + return 'audio'; } return undefined; } @@ -129,7 +131,7 @@ export function projectBridgeMessage( ]; }), ...(source.event.contentParts ? { parts: source.event.contentParts } : {}), - ...(source.platform === 'qq' && source.event.voice + ...((source.platform === 'qq' || source.platform === 'feishu') && source.event.voice ? { voice: { transcript: source.event.voice.transcript, diff --git a/packages/core/src/messaging/dsh-im.ts b/packages/core/src/messaging/dsh-im.ts index d786260f42..c3cf9729d9 100644 --- a/packages/core/src/messaging/dsh-im.ts +++ b/packages/core/src/messaging/dsh-im.ts @@ -316,7 +316,9 @@ const inboundSchema = z .refine( (event) => !event.voice || - ((event.channel === 'weixin' || event.channel === 'qq') && + ((event.channel === 'weixin' || + event.channel === 'qq' || + (event.channel === 'feishu' && event.attachments?.length === 1)) && (!event.attachments?.length || (event.attachments.length === 1 && event.attachments[0]?.mediaType?.startsWith('audio/')))), @@ -357,6 +359,28 @@ function isLarkVideo(event: z.infer, images: boolean): boo ); } +function isLarkVoice(event: z.infer): boolean { + const voice = event.voice; + const item = event.attachments?.[0]; + return ( + voice !== undefined && + event.mentionedAccount && + !!event.reply.parentId && + event.reply.parentId !== event.messageId && + !event.contentParts && + !event.video && + voice.transcript === 'unavailable' && + voice.encodeType === undefined && + voice.sampleRate === undefined && + voice.bitsPerSample === undefined && + event.attachments?.length === 1 && + item !== undefined && + item.id === voice.itemId && + item.messageId === event.reply.parentId && + item.mediaType === 'audio/unknown' + ); +} + function providerFailure(error: unknown): MessagingProviderError { if (error instanceof MessagingProviderError) return error; const code = @@ -885,6 +909,10 @@ export function createDshImProvider( platform === 'feishu' && sourceFiles && info.capabilities.includes('source-video-checked'); + const larkVoices = + platform === 'feishu' && + sourceFiles && + info.capabilities.includes('source-voice-audio-checked'); const qqDirect = platform === 'qq' && info.capabilities.includes('direct-message-consumer'); return host.consumeInbound!(input.accountRef, { @@ -903,9 +931,10 @@ export function createDshImProvider( info.capabilities.includes('source-voice-transcript-checked') ? { sourceVoiceTranscripts: true } : {}), - ...((platform === 'weixin' || platform === 'qq') && - host.fileVersion === 1 && - info.capabilities.includes('source-voice-audio-checked') + ...(larkVoices || + ((platform === 'weixin' || platform === 'qq') && + host.fileVersion === 1 && + info.capabilities.includes('source-voice-audio-checked')) ? { sourceVoiceAudio: true } : {}), ...(larkVideos || @@ -1072,6 +1101,7 @@ export function createDshImProvider( onEvent: async (raw, context) => { const parsed = inboundSchema.parse(raw); const larkVideo = larkVideos && isLarkVideo(parsed, sourceImages); + const larkVoice = larkVoices && isLarkVoice(parsed); const qqAttachments = parsed.attachments?.every((item) => item.mediaType?.startsWith('image/') ? sourceImages @@ -1112,7 +1142,9 @@ export function createDshImProvider( (platform !== 'weixin' || !info.capabilities.includes('source-quote-checked'))) || (parsed.voice && - !info.capabilities.includes('source-voice-transcript-checked')) || + (platform === 'feishu' + ? !larkVoice + : !info.capabilities.includes('source-voice-transcript-checked'))) || (parsed.voice && parsed.attachments?.length && !info.capabilities.includes('source-voice-audio-checked')) || diff --git a/packages/core/src/messaging/outbound.ts b/packages/core/src/messaging/outbound.ts index 25b8988d23..727cb38c87 100644 --- a/packages/core/src/messaging/outbound.ts +++ b/packages/core/src/messaging/outbound.ts @@ -2,6 +2,7 @@ import { readIngest } from './conversation-ingest.js'; import { isChannelNotice } from '../channels/channel-notice.js'; import { sourceContentPurged } from '../purge/fence.js'; import { effectiveDmControls } from './dm-controls.js'; +import { decodeLarkVoice } from '../attachments/lark-audio.js'; import { decodeQqVoice } from '../attachments/qq-audio.js'; import type { ToolApprovalActor } from '../workspaces/tool-approval-actor.js'; import { @@ -1872,7 +1873,9 @@ export function createOutboundMessaging(options: { throw new MessagingError('source-unavailable'); const source = inbound.read(botSlug, sourceEventId); if ( - (source.platform !== 'weixin' && source.platform !== 'qq') || + (source.platform !== 'weixin' && + source.platform !== 'qq' && + source.platform !== 'feishu') || !source.event.voice || !source.event.attachments?.some( (file) => file.id === attachmentId && file.mediaType?.startsWith('audio/'), @@ -1928,7 +1931,11 @@ export function createOutboundMessaging(options: { sourceEventId, attachmentId, source.event.voice, - source.platform === 'qq' ? 'qq-silk-wav-v1' : 'silk-wav-v1', + source.platform === 'qq' + ? 'qq-silk-wav-v1' + : source.platform === 'feishu' + ? 'lark-opus-wav-v1' + : 'silk-wav-v1', ]), ) .digest('hex') @@ -1948,7 +1955,9 @@ export function createOutboundMessaging(options: { const wav = source.platform === 'qq' ? await decodeQqVoice(bytes, combined) - : await decodeWeChatVoice(bytes, source.event.voice!, combined); + : source.platform === 'feishu' + ? await decodeLarkVoice(bytes, combined) + : await decodeWeChatVoice(bytes, source.event.voice!, combined); validate(); return (async function* () { validate(); diff --git a/packages/core/test/channel-bridge-media.test.ts b/packages/core/test/channel-bridge-media.test.ts index 7b369005a7..62b706357d 100644 --- a/packages/core/test/channel-bridge-media.test.ts +++ b/packages/core/test/channel-bridge-media.test.ts @@ -12,6 +12,7 @@ import type { MessagingGrant } from '../src/messaging/outbound.js'; import type { MessagingInboundEvent } from '../src/messaging/provider.js'; import { createAttachmentHttp } from '../src/attachments/http.js'; import { createTempRoot } from './helpers.js'; +import { oggOpus } from './ogg-opus.js'; const png = Buffer.from( 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+a4S8AAAAASUVORK5CYII=', @@ -34,19 +35,31 @@ afterEach(async () => { } }); -async function fixture(inboxOnly = false, twoPaths = false, file = false, video = false) { +async function fixture( + inboxOnly = false, + twoPaths = false, + file = false, + video = false, + audio = false, +) { const mp4 = Buffer.from('000000186674797069736f6d0000020069736f6d69736f32', 'hex'); - const bytes = video ? mp4 : file ? Buffer.from('name,quantity\nalpha,17\n') : png; - const attachment = - file || video - ? { - id: 'c'.repeat(64), - messageId: 'file-parent-message', - resourceKey: 'private-key', - name: video ? 'qa-video.mp4' : 'qa-source.csv', - ...(video ? { mediaType: 'video/unknown' } : {}), - } - : image; + const bytes = audio + ? oggOpus() + : video + ? mp4 + : file + ? Buffer.from('name,quantity\nalpha,17\n') + : png; + const parent = file || video || audio; + const attachment = parent + ? { + id: 'c'.repeat(64), + messageId: 'file-parent-message', + resourceKey: 'private-key', + name: audio ? 'voice.opus' : video ? 'qa-video.mp4' : 'qa-source.csv', + ...(video ? { mediaType: 'video/unknown' } : audio ? { mediaType: 'audio/unknown' } : {}), + } + : image; const home = createTempRoot('bh-channel-media-'); let consumer: Parameters>[1] | undefined; const run = vi.fn(async () => {}); @@ -98,6 +111,7 @@ async function fixture(inboxOnly = false, twoPaths = false, file = false, video 'reply-file-checked', 'source-image-checked', ...(video ? ['source-video-checked'] : []), + ...(audio ? ['source-voice-audio-checked'] : []), ], }), sendChecked: async () => ({ sent: true }), @@ -180,7 +194,10 @@ async function fixture(inboxOnly = false, twoPaths = false, file = false, video ...(video ? { video: { itemId: attachment.id, durationMs: 4000, posterAttachmentId: image.id } } : {}), - ...(file || video + ...(audio + ? { voice: { transcript: 'unavailable' as const, itemId: attachment.id, durationMs: 1000 } } + : {}), + ...(parent ? {} : { contentParts: [ @@ -193,7 +210,7 @@ async function fixture(inboxOnly = false, twoPaths = false, file = false, video messageId: image.messageId, conversationId: 'team', actorId: 'human', - ...(file || video ? { parentId: attachment.messageId } : {}), + ...(parent ? { parentId: attachment.messageId } : {}), }, replay: { kind: 'provider-redelivery', resumeCursor: false, gapPossible: true }, }; @@ -317,6 +334,90 @@ it('projects a checked Lark native video and platform cover once; explicit playb expect(fx.run).toHaveBeenCalledTimes(runs); }); +it('projects a checked Lark native voice and prepares WAV playback only on explicit request', async () => { + const fx = await fixture(false, false, false, false, true); + expect(fx.consumer?.sourceVoiceAudio).toBe(true); + expect(fx.consumer?.sourceVideos).toBeUndefined(); + const messages = fx.core.channels.readMessages(fx.channelId); + expect(messages).toHaveLength(1); + expect(messages[0]?.bridgeMedia).toEqual({ + items: [{ id: fx.attachment.id, kind: 'audio', name: 'voice.opus' }], + voice: { transcript: 'unavailable', durationMs: 1000 }, + }); + expect(JSON.stringify(messages)).not.toContain('private-key'); + expect(fx.download).not.toHaveBeenCalled(); + const before = fx.counts(); + const runs = fx.run.mock.calls.length; + const playback = await fx.playback(); + expect(playback.inline).toBe(true); + expect(playback.ref.mime).toBe('audio/wav'); + const wav = Buffer.from(await new Response(playback.body).arrayBuffer()); + expect(wav.readUInt32LE(24)).toBe(24000); + expect((wav.length - 44) / 48000).toBeGreaterThan(0.9); + const original = await fx.read(); + expect(original.inline).toBe(false); + expect(Buffer.from(await new Response(original.body).arrayBuffer())).toEqual(fx.bytes); + expect(fx.download).toHaveBeenCalledTimes(1); + await fx.restart(); + const cached = await fx.playback(); + await cached.body.cancel(); + expect(fx.download).toHaveBeenCalledTimes(1); + expect(fx.counts()).toEqual(before); + expect(fx.run).toHaveBeenCalledTimes(runs); +}); + +it('refuses a Lark voice that is not the exact quoted parent audio, and keeps a bad codec downloadable', async () => { + const fx = await fixture(false, false, false, false, true); + for (const event of [ + { ...fx.event, voice: { ...fx.event.voice!, itemId: image.id } }, + { ...fx.event, voice: { ...fx.event.voice!, transcript: 'platform' as const } }, + { ...fx.event, voice: { ...fx.event.voice!, encodeType: 6 } }, + { ...fx.event, reply: { ...fx.event.reply, parentId: 'other-parent' } }, + { ...fx.event, attachments: [{ ...fx.attachment, mediaType: 'video/unknown' }] }, + { ...fx.event, mentionedAccount: false }, + ]) { + await expect(fx.consumer!.onEvent(event, { signal: fx.consumer!.signal })).rejects.toThrow(); + } + fx.download.mockImplementation(async function* () { + yield Buffer.from('not ogg opus'); + }); + await expect(fx.playback()).rejects.toThrow('audio-codec-unsupported'); + const original = await fx.read(); + expect(Buffer.from(await new Response(original.body).arrayBuffer()).toString()).toBe( + 'not ogg opus', + ); +}); + +it('requires the Lark voice-audio capability instead of borrowing file qualification', async () => { + const fx = await fixture(false, false, false, false, true); + const info = await fx.service.describeBot('account'); + let input: Parameters>[1] | undefined; + const provider = createDshImProvider({ + ...fx.service, + describeBot: async () => ({ + ...info, + capabilities: info.capabilities.filter((cap) => cap !== 'source-voice-audio-checked'), + }), + consumeInbound: async ( + _id: string, + options: Parameters>[1], + ) => { + input = options; + return () => {}; + }, + })!; + await provider.consume!({ + accountRef: 'account', + fingerprint, + signal: new AbortController().signal, + onEvent: async () => ({ accepted: true }), + }); + expect(input?.sourceVoiceAudio).toBeUndefined(); + await expect(input!.onEvent(fx.event, { signal: input!.signal })).rejects.toThrow( + 'untrusted-source', + ); +}); + it('refuses a Lark video association changed to another parent, primary resource, poster or unsupported content', async () => { const fx = await fixture(false, false, false, true); for (const event of [ diff --git a/packages/core/test/lark-audio.test.ts b/packages/core/test/lark-audio.test.ts new file mode 100644 index 0000000000..4d87779b62 --- /dev/null +++ b/packages/core/test/lark-audio.test.ts @@ -0,0 +1,45 @@ +import { expect, it } from 'vitest'; +import { decodeLarkVoice, oggOpusPackets } from '../src/attachments/lark-audio.js'; +import { MAX_VOICE_INPUT_BYTES } from '../src/attachments/voice-audio.js'; +import { oggOpus as original } from './ogg-opus.js'; + +const signal = () => new AbortController().signal; + +it('decodes Ogg Opus voice to bounded mono 24 kHz PCM WAV', async () => { + const wav = Buffer.from(await decodeLarkVoice(original(), signal())); + expect(wav.subarray(0, 4).toString()).toBe('RIFF'); + expect(wav.subarray(8, 12).toString()).toBe('WAVE'); + expect(wav.readUInt16LE(22)).toBe(1); + expect(wav.readUInt32LE(24)).toBe(24000); + expect(wav.readUInt16LE(34)).toBe(16); + expect(wav.readUInt32LE(40)).toBe(wav.length - 44); + const seconds = (wav.length - 44) / 48000; + expect(seconds).toBeGreaterThan(0.9); + expect(seconds).toBeLessThan(1.05); +}); + +it('splits packets across lacing values and keeps the header facts', () => { + const parsed = oggOpusPackets(original(3)); + expect(parsed).toMatchObject({ channels: 1, preSkip: 312 }); + expect(parsed.packets).toHaveLength(3); +}); + +it('refuses other containers, multiplexed streams, truncated pages and oversized input', async () => { + await expect(decodeLarkVoice(Buffer.from('#!SILK_V3 not ogg'), signal())).rejects.toMatchObject({ + code: 'audio-codec-unsupported', + }); + const mixed = Buffer.concat([original(1), original(1, 8).subarray(0, 60)]); + await expect(decodeLarkVoice(mixed, signal())).rejects.toMatchObject({ + code: 'audio-codec-unsupported', + }); + const bytes = original(); + await expect( + decodeLarkVoice(bytes.subarray(0, bytes.length - 10), signal()), + ).rejects.toMatchObject({ code: 'audio-decode-failed' }); + await expect( + decodeLarkVoice(Buffer.alloc(MAX_VOICE_INPUT_BYTES + 1), signal()), + ).rejects.toMatchObject({ code: 'audio-too-large' }); + const aborted = new AbortController(); + aborted.abort(new Error('stopped')); + await expect(decodeLarkVoice(original(), aborted.signal)).rejects.toThrow('stopped'); +}); diff --git a/packages/core/test/ogg-opus.ts b/packages/core/test/ogg-opus.ts new file mode 100644 index 0000000000..f6b94f712b --- /dev/null +++ b/packages/core/test/ogg-opus.ts @@ -0,0 +1,47 @@ +function crc(page: Uint8Array): number { + let value = 0; + for (const byte of page) { + value ^= byte << 24; + for (let bit = 0; bit < 8; bit++) + value = value & 0x80000000 ? ((value << 1) ^ 0x04c11db7) >>> 0 : (value << 1) >>> 0; + } + return value >>> 0; +} + +function page(packets: Uint8Array[], sequence: number, granule: number, serial = 7): Buffer { + const lacing: number[] = []; + for (const packet of packets) { + for (let left = packet.length; ; left -= 255) { + lacing.push(Math.min(left, 255)); + if (left < 255) break; + } + } + const header = Buffer.alloc(27 + lacing.length); + header.write('OggS', 0); + header[5] = sequence === 0 ? 2 : 0; + header.writeBigUInt64LE(BigInt(granule), 6); + header.writeUInt32LE(serial, 14); + header.writeUInt32LE(sequence, 18); + header[26] = lacing.length; + lacing.forEach((value, index) => (header[27 + index] = value)); + const result = Buffer.concat([header, ...packets]); + result.writeUInt32LE(crc(result), 22); + return result; +} + +export function oggOpus(frames = 50, serial = 7): Buffer { + const head = Buffer.alloc(19); + head.write('OpusHead', 0); + head[8] = 1; + head[9] = 1; + head.writeUInt16LE(312, 10); + head.writeUInt32LE(16000, 12); + const tags = Buffer.alloc(16); + tags.write('OpusTags', 0); + const audio = Array.from({ length: frames }, () => Buffer.from([0xf8, 0xff, 0xfe])); + return Buffer.concat([ + page([head], 0, 0, serial), + page([tags], 1, 0, serial), + page(audio, 2, frames * 960, serial), + ]); +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index e531af4d74..a304ec1696 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -848,6 +848,9 @@ importers: css-tree: specifier: 3.2.1 version: 3.2.1 + opus-decoder: + specifier: 0.7.12 + version: 0.7.12 saxes: specifier: 6.0.0 version: 6.0.0 @@ -4958,6 +4961,9 @@ packages: cpu: [x64] os: [win32] + '@eshaz/web-worker@1.2.2': + resolution: {integrity: sha512-WxXiHFmD9u/owrzempiDlBB1ZYqiLnm9s6aPc8AlFQalq2tKmqdmMr9GXOupDgzXtqnBipj8Un0gkIm7Sjf8mw==} + '@eslint-community/regexpp@4.12.2': resolution: {integrity: sha512-EriSTlt5OC9/7SXkRSCAhfSxxoSUgBm33OH+IkwbdpgoqsSsUg7y3uh+IICI/Qg4BBWr3U2i39RpmycbxMq4ew==} engines: {node: ^12.0.0 || ^14.0.0 || >=16.0.0} @@ -7944,6 +7950,9 @@ packages: '@vscode/ripgrep@1.18.0': resolution: {integrity: sha512-ns5lWe44tSfbTMbVUsyB+I1819PVSw4AdpgK0RNkzfWfwy6+3IUNSxwSrfTno1/oWaS/hERNz+XLWVyga2aJBQ==} + '@wasm-audio-decoders/common@9.0.7': + resolution: {integrity: sha512-WRaUuWSKV7pkttBygml/a6dIEpatq2nnZGFIoPTc5yPLkxL6Wk4YaslPM98OPQvWacvNZ+Py9xROGDtrFBDzag==} + '@webgpu/types@0.1.21': resolution: {integrity: sha512-pUrWq3V5PiSGFLeLxoGqReTZmiiXwY3jRkIG5sLLKjyqNxrwm/04b4nw7LSmGWJcKk59XOM/YRTUwOzo4MMlow==} @@ -11363,6 +11372,9 @@ packages: openapi3-ts@4.6.1: resolution: {integrity: sha512-XW9MOldkhoICNeXVzzmXzmOW5G73ppOEGmh7fLCqHjgfdEYCGGN+00MlVCeUZgovjjfC56j9tvtDt1zGabNjjA==} + opus-decoder@0.7.12: + resolution: {integrity: sha512-6zpHFDi/MhYvMwyYiuJPqJ2lcRkfm7wHblSvT3lUtx/kLvHkunoe/rhTn1dLKxqVGLeQj3gQi7CnR0N2p3RuOg==} + ora@3.4.0: resolution: {integrity: sha512-eNwHudNbO1folBP3JsZ19v9azXWtQZjICdr3Q0TDPIaeBQ3mXLrh54wM+er0+hSp+dWKf+Z8KM58CYzEyIYxYg==} engines: {node: '>=6'} @@ -12409,6 +12421,9 @@ packages: simple-swizzle@0.2.4: resolution: {integrity: sha512-nAu1WFPQSMNr2Zn9PGSZK9AGn4t/y97lEm+MXTtUDwfP0ksAIX4nO+6ruD9Jwut4C49SB1Ws+fbXsm/yScWOHw==} + simple-yenc@1.0.4: + resolution: {integrity: sha512-5gvxpSd79e9a3V4QDYUqnqxeD4HGlhCakVpb6gMnDD7lexJggSBJRBO5h52y/iJrdXRilX9UCuDaIJhSWm5OWw==} + sisteransi@1.0.5: resolution: {integrity: sha512-bLGGlR1QxBcynn2d5YmDX4MGjlZvy2MRBDRNHLJ8VI6l6+9FUiyTFNJ0IveOSP0bcXgVDPRcfGqA0pjaqUpfVg==} @@ -18385,6 +18400,8 @@ snapshots: '@esbuild/win32-x64@0.28.2': optional: true + '@eshaz/web-worker@1.2.2': {} + '@eslint-community/regexpp@4.12.2': {} '@expo-google-fonts/material-symbols@0.4.50': {} @@ -21343,6 +21360,11 @@ snapshots: '@vscode/ripgrep-win32-ia32': 1.18.0 '@vscode/ripgrep-win32-x64': 1.18.0 + '@wasm-audio-decoders/common@9.0.7': + dependencies: + '@eshaz/web-worker': 1.2.2 + simple-yenc: 1.0.4 + '@webgpu/types@0.1.21': {} '@xmldom/xmldom@0.7.13': {} @@ -25292,6 +25314,10 @@ snapshots: dependencies: yaml: 2.9.1 + opus-decoder@0.7.12: + dependencies: + '@wasm-audio-decoders/common': 9.0.7 + ora@3.4.0: dependencies: chalk: 2.4.2 @@ -26682,6 +26708,8 @@ snapshots: dependencies: is-arrayish: 0.3.4 + simple-yenc@1.0.4: {} + sisteransi@1.0.5: {} sitemap@9.0.1: diff --git a/scripts/dev-im-provider.mjs b/scripts/dev-im-provider.mjs index 09b8d21ae8..64ff825338 100644 --- a/scripts/dev-im-provider.mjs +++ b/scripts/dev-im-provider.mjs @@ -5,11 +5,11 @@ import { join } from 'node:path'; export const qualifiedImProvider = Object.freeze({ package: '@xmanrui/dsh-im', packageVersion: '4.32.0', - source: 'e655636c061fa0aed8c77c23f7e87cb5abf79b65', - spec: 'github:DoodleBears/dsh-im#e655636c061fa0aed8c77c23f7e87cb5abf79b65', + source: 'bb0c7956896f5f5845950a0707837cd7938a11b3', + spec: 'github:DoodleBears/dsh-im#bb0c7956896f5f5845950a0707837cd7938a11b3', dsh: '0.2.1-rc.2', runtimeFiles: 425, - runtimeSha256: '7b9cbc1706ed42a114d10d106ddf41d1fcbbc3f93dcfbe358950802e695f1647', + runtimeSha256: 'b1df502e022b25e474a691b8c99517f7f6a7222b308c7f3fe2aff23232540751', }); export function withQualifiedImProvider(manifest) { diff --git a/scripts/product-artifacts.mjs b/scripts/product-artifacts.mjs index 14d89c1999..bf94216799 100644 --- a/scripts/product-artifacts.mjs +++ b/scripts/product-artifacts.mjs @@ -21,16 +21,16 @@ export const productDshVersions = Object.freeze(['0.2.0-rc.2', '0.2.1-rc.2', pro export const productImProvider = Object.freeze({ name: '@botharness/im-provider', - version: '4.32.0-botharness.26', + version: '4.32.0-botharness.27', sourceManifestSha256: '3331e8f771a91abf396eb4e7ed5c0acc4c3b504cdccc7d715e37062708384cc2', sourceLockSha256: '3476c34c7d4515000d2617beaf2568b2a794b60de9b350cd8eafb7a2bf21adef', upstream: Object.freeze({ package: '@xmanrui/dsh-im', packageVersion: '4.32.0', - source: 'e655636c061fa0aed8c77c23f7e87cb5abf79b65', + source: 'bb0c7956896f5f5845950a0707837cd7938a11b3', dsh: '0.2.0-rc.2', runtimeFiles: 425, - runtimeSha256: '7b9cbc1706ed42a114d10d106ddf41d1fcbbc3f93dcfbe358950802e695f1647', + runtimeSha256: 'b1df502e022b25e474a691b8c99517f7f6a7222b308c7f3fe2aff23232540751', }), }); diff --git a/scripts/test/product-artifacts.test.mjs b/scripts/test/product-artifacts.test.mjs index 57bb5f6ad7..9becdfc2b2 100644 --- a/scripts/test/product-artifacts.test.mjs +++ b/scripts/test/product-artifacts.test.mjs @@ -324,7 +324,7 @@ describe('release composition', () => { expect(() => verifyProductComposition(entries)).toThrow('conflicts'); }); it('keeps product provenance independent when the development Provider selection changes', () => { - expect(productImProvider.upstream.source).toBe('e655636c061fa0aed8c77c23f7e87cb5abf79b65'); + expect(productImProvider.upstream.source).toBe('bb0c7956896f5f5845950a0707837cd7938a11b3'); expect(productImProvider.upstream.dsh).toBe('0.2.0-rc.2'); }); @@ -339,7 +339,7 @@ describe('release composition', () => { '@botharness/computer': '0.0.0-test.823', '@botharness/core': '0.0.0-test.823', '@botharness/ui': '0.0.0-test.823', - '@botharness/im-provider': '4.32.0-botharness.26', + '@botharness/im-provider': '4.32.0-botharness.27', }); expect(release.optionalDependencies).toEqual( Object.fromEntries(cloudflaredPackageNames.map((name) => [name, '0.0.0-test.823'])), @@ -377,11 +377,11 @@ describe('release composition', () => { describe('product-managed Provider updates', () => { it('reports the running artifact without starting any independent check or installation job', () => { - expect(productManagedUpdate('update.status', {}, undefined, '4.32.0-botharness.26')).toEqual({ + expect(productManagedUpdate('update.status', {}, undefined, '4.32.0-botharness.27')).toEqual({ ok: true, value: { - runningVersion: '4.32.0-botharness.26', - installedVersion: '4.32.0-botharness.26', + runningVersion: '4.32.0-botharness.27', + installedVersion: '4.32.0-botharness.27', latestVersion: null, canInstall: false, sourceInstall: false, @@ -405,7 +405,7 @@ describe('product-managed Provider updates', () => { 'refuses %s so upstream update paths cannot replace the qualified artifact', (endpoint, payload) => { expect( - productManagedUpdate(endpoint, payload, undefined, '4.32.0-botharness.26'), + productManagedUpdate(endpoint, payload, undefined, '4.32.0-botharness.27'), ).toMatchObject({ ok: false, error: { code: 'product-managed' } }); }, ); @@ -413,7 +413,7 @@ describe('product-managed Provider updates', () => { const controller = new AbortController(); controller.abort(); expect( - productManagedUpdate('update.status', {}, controller.signal, '4.32.0-botharness.26'), + productManagedUpdate('update.status', {}, controller.signal, '4.32.0-botharness.27'), ).toMatchObject({ ok: false, error: { code: 'cancelled' } }); }); }); diff --git a/tsdown.config.ts b/tsdown.config.ts index 5b0072cf6d..c2bfa13371 100644 --- a/tsdown.config.ts +++ b/tsdown.config.ts @@ -119,7 +119,7 @@ export const coreCliBundleOptions: UserConfig = { platform: 'node', dts: false, clean: true, - external: ['@resvg/resvg-js', 'silk-wasm', 'css-tree'], + external: ['@resvg/resvg-js', 'silk-wasm', 'opus-decoder', 'css-tree'], deps: { alwaysBundle: [/^@deepseek-ai\//] }, }; @@ -220,6 +220,7 @@ export default defineConfig([ external: [ '@resvg/resvg-js', 'silk-wasm', + 'opus-decoder', 'css-tree', '@deepseek-ai/dsh-agent', '@deepseek-ai/dsh-llm', From 7ed18cba0b8adede609a995a3db825e8df924c65 Mon Sep 17 00:00:00 2001 From: DoodleBear Date: Tue, 3 Nov 2026 17:09:27 +0900 Subject: [PATCH 2/3] fix(client): accept Lark voice media in Channel bubbles Real QA showed the Client parser dropped bridged voice media from Lark, leaving only the request text. Also name Opus in the playback hint. Refs #1023 Co-Authored-By: Claude Opus 5.5 (1M context) --- packages/client/src/client/bridge.ts | 2 +- packages/client/src/client/locale.ts | 5 ++-- packages/client/test/bridge-audio.test.ts | 28 +++++++++++++++++++++++ 3 files changed, 32 insertions(+), 3 deletions(-) diff --git a/packages/client/src/client/bridge.ts b/packages/client/src/client/bridge.ts index 5b71652eb0..bf38d034c8 100644 --- a/packages/client/src/client/bridge.ts +++ b/packages/client/src/client/bridge.ts @@ -1389,7 +1389,7 @@ export function parseChannelMessage(value: unknown): ChannelMessage | undefined if (media['voice'] !== undefined) { const voice = asRecord(media['voice']); if ( - bridgeOrigin.platform !== 'qq' || + (bridgeOrigin.platform !== 'qq' && bridgeOrigin.platform !== 'feishu') || !voice || Object.keys(voice).some((key) => !['transcript', 'durationMs'].includes(key)) || (voice['transcript'] !== 'platform' && voice['transcript'] !== 'unavailable') || diff --git a/packages/client/src/client/locale.ts b/packages/client/src/client/locale.ts index c58a2a2f98..a4a1ba1587 100644 --- a/packages/client/src/client/locale.ts +++ b/packages/client/src/client/locale.ts @@ -2616,7 +2616,8 @@ export const zh = { 'im.voiceAudioTooLarge': '语音超过播放大小限制,仍可尝试下载原音。', 'im.voiceAudioInterrupted': '播放准备已中断,请重试。', 'im.voiceAudioPlayer': '语音播放器', - 'im.voiceAudioHint': '支持的 SILK 语音转换为 24 kHz WAV,仅用于播放或处理,不进行语音识别。', + 'im.voiceAudioHint': + '支持的 SILK 或 Opus 语音转换为 24 kHz WAV,仅用于播放或处理,不进行语音识别。', 'im.voiceAudioUnavailable': '此音频暂不能播放:可能编码不支持、解码失败或超过处理限制。仍可下载原始语音;需要内容时请发送文字。', 'im.voiceDownloadOriginal': '下载原始语音', @@ -7272,7 +7273,7 @@ My relationship with you, established lore and topics to avoid: [fill in]. 'im.voiceAudioInterrupted': 'Playback preparation was interrupted. Please retry.', 'im.voiceAudioPlayer': 'Voice player', 'im.voiceAudioHint': - 'Supported SILK is converted to 24 kHz WAV for playback or processing, without speech recognition.', + 'Supported SILK or Opus voice is converted to 24 kHz WAV for playback or processing, without speech recognition.', 'im.voiceAudioUnavailable': 'Audio playback is unavailable: unsupported encoding, decode failure or processing limit. You can still download the original; send text when its content is needed.', 'im.voiceDownloadOriginal': 'Download original voice', diff --git a/packages/client/test/bridge-audio.test.ts b/packages/client/test/bridge-audio.test.ts index b9917c8eb1..13890a2051 100644 --- a/packages/client/test/bridge-audio.test.ts +++ b/packages/client/test/bridge-audio.test.ts @@ -161,3 +161,31 @@ it('unmount aborts pending playback and discards a late response', async () => { vi.unstubAllGlobals(); } }); + +it('accepts Lark voice media and refuses voice evidence from platforms without a checked audio path', () => { + const message = (platform: string) => + parseChannelMessage({ + id: 'source', + at: '2026-11-03T00:00:00.000Z', + body: 'please listen', + format: 'text', + author: { kind: 'bridged', source: 'Human' }, + bridgeOrigin: { + sourceEventId: 'source', + platform, + conversationId: 'group', + conversationName: 'Group', + messageId: 'native-message', + senderId: 'human', + }, + bridgeMedia: { + voice: { transcript: 'unavailable', durationMs: 4916 }, + items: [{ kind: 'audio', id: 'a'.repeat(64), name: 'voice.opus' }], + }, + }); + expect(message('feishu')?.bridgeMedia).toEqual({ + voice: { transcript: 'unavailable', durationMs: 4916 }, + items: [{ kind: 'audio', id: 'a'.repeat(64), name: 'voice.opus' }], + }); + expect(message('discord')).toBeUndefined(); +}); From 121f7e2c2117deb91bee2a2809226606855fe828 Mon Sep 17 00:00:00 2001 From: DoodleBear Date: Tue, 3 Nov 2026 17:13:35 +0900 Subject: [PATCH 3/3] docs: record real Lark voice playback qualification Refs #1023 Co-Authored-By: Claude Opus 5.5 (1M context) --- ...dge-media-uses-channel-source-authority.md | 2 ++ .../verification/lark-1023-channel-audio.md | 24 +++++++++++++++++++ 2 files changed, 26 insertions(+) create mode 100644 docs/dev/verification/lark-1023-channel-audio.md diff --git a/docs/adr/0137-human-bridge-media-uses-channel-source-authority.md b/docs/adr/0137-human-bridge-media-uses-channel-source-authority.md index 9393a00c3e..44da73f38e 100644 --- a/docs/adr/0137-human-bridge-media-uses-channel-source-authority.md +++ b/docs/adr/0137-human-bridge-media-uses-channel-source-authority.md @@ -33,6 +33,8 @@ The #1023 first video tracer extends this application-defined contract to a chec Human preparation reuses the existing authenticated attachment route, source authority and bounded acquisition owner. `representation=playback` serves the unchanged original only when byte sniffing identifies MP4; the browser independently verifies decoding. No transcoder, generated poster or model transcript is introduced. The original remains independently downloadable even when its playback format is refused. Covers are acquired only after preparation, through their own exact source descriptor; missing or failed covers do not block video. Browser resources cancel and release URLs on unmount or authority changes. The CLI requests the same representation explicitly. Mobile currently offers the original as a file action and does not automatically load the video cover. Audio and direct native media messages remain subsequent qualification scopes. +The #1023 audio slice applies the same contract to a native Lark `audio` parent. Provider `sourceVoiceAudio` retains its exact resource as one `audio/unknown` attachment plus a voice descriptor with `transcript: 'unavailable'` and optional platform duration. Human preparation reuses the existing voice `prepareAudio` owner: the Host decodes Ogg Opus to bounded 24 kHz WAV, as it already does for WeChat and QQ SILK. Speech recognition is not added. Qualification is recorded in [the audio checkpoint](../dev/verification/lark-1023-channel-audio.md). + The candidate's automated regressions do not establish native intake, rendering, playback or installed-package qualification. Those gates and the required first-slice Human feedback remain tracked in [#1023](https://github.com/BotHarness/DeepSeekBot/issues/1023). Opening a Bot Inbox source inspector must not bypass the Lark video's explicit acquisition policy. That existing inspector keeps original-download actions for video and cover, but excludes both from its legacy automatic image/video preview loop. Existing qualified WeChat video previews and unrelated Lark images keep their own presentation behavior. diff --git a/docs/dev/verification/lark-1023-channel-audio.md b/docs/dev/verification/lark-1023-channel-audio.md new file mode 100644 index 0000000000..f3c0633203 --- /dev/null +++ b/docs/dev/verification/lark-1023-channel-audio.md @@ -0,0 +1,24 @@ +# Lark parent-voice tracer — #1023 audio slice + +## Scope + +The audio slice of [#1023](https://github.com/BotHarness/DeepSeekBot/issues/1023), after the video slice ([#1950](https://github.com/BotHarness/DeepSeekBot/pull/1950)). A native text request that mentions the receiving App and replies to a native Lark `audio` message carries that parent's exact voice descriptor. Direct native audio messages (without a quoting request) and speech recognition remain outside this slice. + +Provider: [dsh-im#35](https://github.com/DoodleBears/dsh-im/pull/35), source `bb0c7956896f5f5845950a0707837cd7938a11b3`, 425 runtime files with SHA-256 `b1df502e022b25e474a691b8c99517f7f6a7222b308c7f3fe2aff23232540751`. + +## Implemented path + +Provider `sourceVoiceAudio` opt-in → one `audio/unknown` attachment `voice.opus` bound to the parent message and `voice: { transcript: 'unavailable', itemId, durationMs? }` → Core accepts it only as the exact quoted parent (`isLarkVoice`) and with `source-voice-audio-checked` → Channel projection exposes an `audio` item and the platform duration → explicit **Prepare playback** → existing `prepareAudio` owner reads the original under current source authority and decodes Ogg Opus to 24 kHz mono WAV in a bounded worker (`opus-decoder`, 1 MiB input, 12 MiB PCM, 10 s) → browser playback. The original stays downloadable; other containers are refused with `audio-codec-unsupported`. + +## Native qualification — 2026-11-03 JST + +Isolated Host on DSH `0.2.1-rc.3`, model DeepSeek V4.1 Flash, app `BotHarness Lark QA1`, group "BotHarness QA1 Test" (test account and Bot only). A 4.9 s synthetic Opus voice was sent as a native `audio` message through the Bot API; a real Human reply in the Lark web client quoted it and mentioned the Bot through the native @ picker. + +- Intake: the Channel message carried one `audio` item `voice.opus` and `voice: { transcript: 'unavailable', durationMs: 4916 }`. No attachment file existed on the Host before preparation, and the Web Client sent no attachment request before **Prepare playback**. +- Playback: after preparation Chromium played the WAV (`duration` 4.907 s, time advanced 1.3 s in 1.5 s, seek to 3.5 s, no media error) in light and dark. +- CLI: `channel-media-download` returned the original byte-identical to the sent file (`156874cb…`, 43,237 bytes); with `--representation playback` it returned `audio/wav` (235,580 bytes). +- The Bot replied once in Lark and said no transcript was available. + +The run found one defect, now fixed: the Client parser accepted bridged voice only from QQ, so the Lark bubble showed text without the audio card. + +Not exercised: a voice recorded by a Human in a Lark client (the browser tool cannot record audio), the installed product and the mobile app.