组织级测试体系 - Systematic Thinking for Testing

分享到社交媒体：

Note: You can find the English version after the Chinese one.
(中文版文末有对应演讲视频)

【中文版 Chinese version】

00 引言

1. 三个层次聊测试体系

测试人员缺乏体系化思维？
新建产品团队或者新启项目，如何搭建质量保障体系？

大家都接触过不计其数的测试、质量方面的文章或者培训课程，内容不乏测试实践、技术相关，但是却很难构建自己的测试体系。基于很多朋友类似的困惑，结合本人多年的团队实践和咨询经验，从基础篇、进阶篇和高级篇三个不同的层次来跟大家探讨测试体系化思维的构建。

2. 基础篇和进阶篇内容回顾

2022年1月发布了《构建测试的体系化思维（基础篇）》（后文简称《基础篇》），从测试的五个基本职责出发，围绕这五个基本职责介绍了相应的实践活动和方法集。

理解和澄清业务需求
制定策略并设计测试
实现和执行测试
缺陷管理与分析
质量反馈与风险识别

3月发布了《构建测试的体系化思维（进阶篇）》（后文简称《进阶篇》），跳出测试，从软件质量的角度来看体系化思维的构建。

从测试到质量：跳出测试，从质量维度来看测试的体系化建设。
质量保障与质量内建：质量是产品与生俱来的，质量内建才是保障质量的根本。
质量内建深入实践：质量内建如何落地是大家最为关注的问题，深入质量内建实践介绍，供大家落地实践参考。

3. 高级篇内容概要

本文为高级篇，将从组织级别来看测试体系化思维的构建。

从团队到整个组织：跳出产品/项目团队，从整个组织范围来看测试的体系化建设。
组织级测试体系图谱：基于“一个中心，四个方向”图谱，全面探讨组织级测试体系建设。
组织级质量赋能：根据测试体系图谱，从组织级层面对整个组织进行质量赋能。

01 从团队到组织

1. 为什么要从团队到整个组织？

在《基础篇》中聊到的测试基本职责和《进阶篇》中聊到的质量内建，主要还是在团队/项目范围之内。但是，要真正做好测试体系建设或者说系统性地思考和开展质量保障工作，团队的能力还是很有限，比如，下面这些场景有没有觉得似曾相识？

我所在的组织有很多绩效考核指标，工作靠绩效驱动，感觉有些工作并不能带来价值
我在一个独立的测试部门，跟开发团队的绩效是不一致/矛盾的
业务部门离我很远，所有开发和测试工作只能根据需求文档开展
组织的各种评审流程很严格，对文档要求很高，很多时间花在了写文档、走评审流程上
所开发的软件产品对上下游都有依赖，测试数据的准备和测试环境的就绪需要很多等待
我不清楚其他团队使用的工具、测试技术等具体细节
……

上面这些场景中的问题，都是很难在某个团队内部解决的。因此，从组织范围来看测试体系的建设，才是我们构建测试体系化思维的终极目标。

2. 组织级测试体系需要关注什么？

质量保障相关的人、流程、技术等放到组织层面，就是组织级测试体系需要关注的。通常有如下两个方面的内容：

跟质量和测试密切相关的，包括质量目标、流程规范、测试策略、实践标准等；
看似跟质量和测试没有直接的关系，但是对质量保障工作的开展有着非常关键的影响，如组织结构、企业文化、各部门之间的合作方式等。

02 组织级测试体系图谱

组织级测试体系必然比团队级要复杂得多，为了更好地理解和讨论，将组织级测试体系需要关注的方方面面组织成如下图所示图谱：

该图谱由“一个中心，四个方向”构成，要构建完备的组织级测试体系，建议围绕“一个中心”、向四个方向发力：

一个中心：核心价值观
四个方向：高效率协同、标准化指导、规范化实施、自动化支撑

1. 核心价值观

核心价值观是一个组织的文化理念，深刻地影响着组织内每个成员的行为，测试体系的构建需要以核心价值观为指导。

1.1 价值驱动 vs. 指标驱动

绝大部分组织都会采用度量指标来对每个人的工作进行考核，因为这是一种比较简单的衡量所做工作好与坏的方法，但这种方法也是相当粗暴的，因为“不是所有能测量的东西都重要，也不是所有重要的东西都可测量”。

过于看着指标，就会形成指标驱动，工作是为了满足某种指标。测量什么，就一定能得到什么。一旦制定某个度量指标，员工一定会在工作中想尽办法去满足指标，而真正重要的工作内容是否完成就很难说了。这种指标驱动的工作方式，一定会忽略工作背后真正的价值。这是非常可怕的！在《指标陷阱》一书开篇就举了两个非常经典的例子：警察破案率、妇产科医生的手术成功率。

我们来看一个软件开发中以指标驱动的例子。在《速度（Velocity）不背这个锅》一文中提到的“速度驱动一切”的误区，就是软件开发中错误地以指标（开发速度）来驱动开发的实例。如下图所示：

相信各位朋友要举出软件开发中指标驱动的例子，一定是信手拈来，此处不再多述。

那么，是不是就没必要度量了呢？也不是，适当的度量还是很有必要的。度量可以作为团队改进的牵引，帮助团队提高。

但是，工作不可以以任何度量指标为目标，不能由度量指标驱动，而是要以价值驱动。软件开发是需要交付能够带来业务价值的、高质量的软件，围绕这个价值目标所做的工作才是有意义的。因此，软件的质量保障、软件测试也要以业务价值驱动。关于这部分内容，请参考文章《业务价值驱动的测试》。

从组织层面来讲，需要构建价值驱动的文化，引导员工更多地关注价值，弱化度量指标带来的不良影响，以防掉入指标陷阱。

1.2 鼓励创新 vs. 追责文化

绩效考核的一个直接后果就是追责，追责文化是跟绩效指标紧密关联的。绩效指标没有达到，甚至是出现意外，是采取追责还是改进的措施，将直接影响到员工的行为、态度和解决问题的方法。比如：

线上出现严重故障，最终会调查追责到人，并施以惩罚；还是会分析故障原因，找出后续如何避免的改进举措呢？

如果是前者，一旦出现问题，所有相关人员会提心吊胆，很难客观地分析和解决问题，反而会有某些隐瞒因素存在，导致下次可能还会重复出现类似问题。关于后者，之前在文章《缺陷分析如何帮助质量内建》中有关于故障/缺陷分析的详细实践，这里想要强调一点的是缺陷分析，分析到问题出现的客观原因即可，无需追责到人。

一个非常典型的不追责例子是敏捷回顾会议所遵循的最高指导原则：

"Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand."
（“无论我们揭示了什么，我们必须理解并真正相信：考虑到当时的已知情况、每个人的技能和能力、可用资源和情境，每个人都做到了最好。”）

-- Norm Kerth, Project Retrospectives: A Handbook for Team Review

不追责的另一面是鼓励创新。成功来自于不停地试错，要想把事情做好，需要不断地尝试，经历多次的不成功，并且在一次次失败中总结经验教训，从而获得进步。例如，高科技公司谷歌和亚马逊，都是非常鼓励创新的。

组织要宣扬这种创新文化，给予员工适当的创新空间，鼓励大家积极地尝试新点子，以不断改进软件开发流程，提高软件交付质量。

1.3 全员负责 vs. 独立割裂

对于软件开发团队来说，“团队对质量负责”几乎已经是家喻户晓了。但是，从组织级来看，团队该怎么定义？道理都明白，具体到实施层面，又是怎么做的呢？

据我所知的现实情况，部门与部门之间、团队不同角色之间还是比较割裂的，还有许多人认为质量主要是测试（部门/团队/人员）的事情，而业务负责业务需求、开发负责编码、运维负责线上就行了。

之前写过几篇团队对质量负责的文章，欢迎移步阅读：

从组织角度来看，需要明确质量目标，并让全体成员知晓，以质量目标驱动各个职能部门间的合作，增强全员对质量负责的意识。

2. 高效率协同

高效率协同强调的是组织结构形式，以及组织内各部分的协同方式，这是组织级测试体系构建非常关键的一部分。

2.1 组织结构

设计系统的结构受制于产生这些设计的组织的沟通结构。

—— M. Conway（马尔文·康威）

马尔文·康威1967年提出的康威定律（康威法则, Conway's Law），即系统设计本质上反映了企业的组织结构。

软件开发是一项复杂的社会活动，需要业务、开发、测试和运维以及团队与团队之间等充分地沟通协作才能实现，根据康威定律，需要对组织结构进行相应地调整以适应软件研发的需要。

而传统的企业通常都有着非常厚重的部门墙，业务和科技之间，开发和测试、以及运维之间，都分别属于不同的部门，并且这种部门墙非常难以打破。这无疑是非常大的矛盾之处。

例如，传统企业通常都有独立而庞大的测试部门/测试团队，在独立的测试阶段来承接测试工作；在新形势下要求持续测试，不再有非常独立的测试阶段，测试部门该如何调整以更好地与研发进行高效合作是备受关注的问题，也是组织建立完善的测试体系需要解决的问题。

该怎么解决？比较常见的有两种方式：

i. 保持原有独立测试部门，测试人员归属于测试部门管理，但是派驻到研发团队，在研发团队内实施测试工作；
ii. 不再设有测试部门，测试人员全部打散，并入研发中心，跟研发团队进行深度融合。

上面哪种方式比较好？很难说。

从形式上看，完成融合的第二种方式看起来是较好的，能够实现开发和测试更深入的融合，但是这对团队也是有很高要求的，需要在测试左移、团队对质量负责等方面都做的足够好，才能对于质量有让人放心的保障。因此，对于质量要求非常高的行业，比如金融领域，比较容易接受的可能是第一种方式，一是因为传统的部门墙打破实在太难，还有很重要的一方面是还有独立的测试部门对质量“把关”，似乎能让大家更放心。

至于自己的组织适合哪一种形式，还是需要组织自己去摸索。

2.2 团队拓扑

组织应该被视为一个复杂的具有适应性的有机体，而非一个机械的线性系统。

—— Naomi Stanford，摘自《组织设计指南》（Guide to Organization Design)

《高效能团队模式》一书中指出组织结构的陷阱：传统的组织结构图是相对静态的，并不能帮助我们理解组织中真实的沟通模式，在实际的软件开发过程中可能有很多不依赖于组织结构图中所体现的沟通存在，这种适应软件开发的沟通结构是更为关键而有价值的。因此，提出团队拓扑的概念。

（《高效能团队模式》作者总结的团队拓扑Team Topologies）

团队拓扑方法围绕企业软件交付的有效团队结构，带来了一种全新的思维方式。提供了四类基本的团队类型：流动式团队、平台团队、赋能团队和复杂子系统团队，以及三种团队核心交互模式：协作模式、服务模式和促进模式。强调了在静态的组织结构（团队结构）之外，更多地关注实际的沟通路径，针对技术组织补充了传统组织设计中所缺少的动态和感知方面的能力。

回顾前面提到的测试部门如何组织的问题，基于团队拓扑思想，我们可能会想到不一样的解决方案，加强测试跟各个部门/角色的交互才是关键所在，需要根据产品开发需求构建灵活适应的团队拓扑结构，具体的组织形式或许不是最重要的。

2.3 沟通协作

团队拓扑强调了组织沟通结构的重要性，在满足沟通模式的团队构建好之后，是不是就一定能实现相应的沟通需求呢？也不见得。

例如，下面的一些情况可能大家都或多或少经历或者听说过：

有些业务目标、质量目标可能只是“高层”领导人员清楚，没有对全组织、全团队透明；
有的企业，业务和研发团队坐在一起，但是沟通方式跟原来没有区别，还是通过需求文档来进行沟通；
有的团队，测试和开发同属于一个团队，但是测试不会参与开发的技术讨论，开发有技术方面更新不会告知测试人员，开发不会参与测试策略和测试计划的制定等；
同一个产品或者项目的不同小团队各自为伍，没有知识和信息共享，导致重复造轮子的事情时有发生……

类似的情况可能还有很多。这些都是形式上貌似没有问题，但实质上缺乏有效的沟通协作。

关于沟通，首先要重视沟通的内容，尽量让信息在整个组织或者整个团队内透明，尤其是关于愿景、目标和风险类的信息，可以增强成员使命感和主观能动性。我在《敏捷测试的指导性原则》里介绍了高效合作的团队特点，在《警惕团队“蘑菇种植”》中也通过一些事例分享了信息共享的重要性。

另一方面，对于沟通的形式也要注意，不同的沟通形式所能起到的效果也是截然不同的。通常，面对面的沟通是最为有效的方式。如下图所示：

协作建立在充分沟通的前提上，一旦沟通到位了，协作起来就会顺畅很多。

3. 标准化指导

之前写过一篇文章《敏捷团队的质量保障赋能》，文中通过对Google、Amazon和Facebook等大公司的软件开发交付流程的分析，总结了团队质量保障赋能需要做到全流程的标准化。

通过标准化工作方式的指导，让团队能更好地做到每个环节的质量保障，更好地实现团队为质量负责。标准化一定要注意不要只流落于书面形式，参考《敏捷团队的质量保障赋能》中的“事实标准与书面标准”。

3.1 研发流程的标准化

测试和开发是密不可分的，测试策略受到不同开发流程的影响。

不管是传统瀑布式的开发流程，还是迭代式的敏捷开发流程，或者两者相结合的开发方式，都需要将相应地流程策略标准化下来，每个环节的活动有哪些，不同人员的职责分别是什么，都需要在组织级有清晰的定义。

在标准化的研发流程指导性，团队如果发现某个环节成为瓶颈，一定要详细分析原因，如果是流程已经不适应新的团队现状，也是需要调整和改进的。

3.2 测试策略的标准化

针对不同的开发流程，需要有对应的组织级标准化测试策略。组织级测试策略是个统一的指导方向，不同的产品线和不同的项目团队也需要有自主调整的空间，以更好地定制化适配的策略。

除了需要给团队自主调整策略的空间，还有一点要强调的是策略不是一成不变的，需要根据不同时期的质量风险和团队状态进行适配和调整，应该是演进式的。

关于测试策略，强烈推荐参考《一页纸测试策略》，并根据所在组织自己的特点来定制化。

3.3 质量目标和度量策略标准化

清晰的质量目标能够更好地驱动组织级测试体系的构建。组织级的质量目标，一方面是纯系统使用方面的质量要求，更为关键的另一方面，应该是跟业务目标紧密关联的，是需要以业务价值来驱动的。关于业务价值与测试，可以参考我的下列文章：

《业务价值驱动的测试》
《敏捷测试如何优化业务价值》

谈到质量目标，势必需要相应的度量指标和度量策略，组织级需要对质量目标定义相应的指标，并且通过组织统一的度量策略来衡量。度量，特别需要关注的是不可以只看单一指标，而应该是综合多个指标衡量；也不适合将指标作为产品线/团队横向比较的标准，指标应该是指导产品线/团队自身持续改进的尺子，非常同意Thoughtworks同事伍斌道长在他的文章《度量就是为了识别价值流最大瓶颈》中写的：

“在敏捷IT研发交付中，度量的作用，就好比是在识别价值流中最大的堵塞点，以便在“价值准、流速快、质量好”这3个维度中，识别端到端价值流最大瓶颈（以及方向错误），并将其作为下一步改进点进行改进，以最大化改进成效。”

作者：吾真本
链接：https://www.jianshu.com/p/91eaf583ca3b
来源：简书
著作权归作者所有。商业转载请联系作者获得授权，非商业转载请注明出处。

关于质量度量，Thoughtworks同事于晓南从定性和定量两个角度进行过非常详细的分享，并且撰写过一系列相关文章，欢迎参考她的《质量度量文集》。

4. 规范化实施

在标准化策略的指导下，如何保证每个实践活动都能做到位是个比较难的问题：有的实践可能一开始就实施错了；或者刚开始是正确的，但是过了一段时间，实践就变味儿了……

规范化的实施方案是对标准化策略落地实施的保障。组织级需要从以下几个方面考虑各个实践活动的有效落地实施：定义实践活动规范、沉淀新的优秀实践、定制化成熟度模型、定期治理与持续改进。

4.1 定义实践活动规范

对每个实践活动进行清晰的定义，包括实践的目标、适用条件、参与角色、输入输出等，最好用实例化的方式详细介绍整个实践活动的内容。可以采用画布的方式对前面几项列举介绍，更加清晰直观。

定义好的实践规范是团队实施落地的参考，在团队实践过程中，也鼓励团队根据实践经验去完善每个实践活动，实现持续地优化。

4.2 沉淀新的优秀实践

团队在实践过程中，也会摸索出一些新的实践活动，建议将新的实践活动分享给更多的团队，推荐大家试用，一旦得到多个团队的认可，可以将这个实践活动也规范化，沉淀下来，存入组织级实践库中。

同时，对于一些不再适用的实践活动，也要进行淘汰和剔除。

4.3 定制化成熟度模型

基于实践标准的成熟度模型，可以评估团队每个阶段的实践实施状况，并制定相应的改进举措，以帮助团队的不断成长。组织级需要制定指导性的成熟度模型，供每个团队去定制化，以牵引团队的改进。这块内容欢迎参考我的文章《测试成熟度评估》。

特别要强调一点：成熟度模型的目的不是用来评估，主要是为了指导改进；成熟度如果要跟团队绩效关联，务必谨慎，不然就会陷入指标陷阱。

4.4 定期治理与持续改进

光有成熟度模型没用，要能指导改进，定期治理很重要。组织级层面需要制定定义质量治理策略，落到每个团队来讲，就是可以根据成熟度模型，定期对过去一段时间实践落地情况进行回顾，对现状进行评估，根据回顾和评估结果，并结合当下的质量目标，制定新的改进举措。

比如，可以鼓励团队根据迭代周期、发布周期等进行定期回顾和治理；也可以根据改进举措的粒度大小，对不同举措进行不同周期的治理和改进。

总之，组织级要有定期治理和持续改进的意识，结合成熟度模型，鼓励团队实现持续改进。

5. 自动化支撑

强有力的基础设施能够对质量实践活动落地实施提供自动化支撑，同步提升质量与效能。

从测试体系构建的角度来讲，组织级需要全盘考虑几个领域的自动化支撑：

流程管理：全生命周期的软件交付流程，通常需要有流水线的支撑
测试框架：各种自动化测试框架，以及与持续集成流水线的关联
数据分析：可视化各种数据及其分析结果，包括日志信息和度量指标数据等
监控预警：实时监控团队和产品状态，对质量风险和严重缺陷进行智能预测和告警

只要聊到效能，大家脑海里就会冒出各种效能支持平台和工具，这部分内容是非常熟悉的，无需多述……不过，鉴于大家对工具平台的热衷程度，非常有必要提醒几点：

强调工具平台的使用，但人员缺乏相应的赋能，不清楚工具平台背后的原理，从而无法高效使用。平台厂商和采购平台的企业都有人跟我聊过这个现象，说明不是个例。工具平台再好，也只有用对了才能事半功倍，不然就是个负担。
标准规范停留在书面上，根本没有落地实施。这就是前面提到的《敏捷团队的质量保障赋能》中的“事实标准与书面标准”，而工具平台是有助于将书面标准变成事实标准的。
不同的团队采用不同的工具框架，组织没有统一工具平台的使用。这将不利于不同团队间的协作和集成，不利于组织级数据的收集，会增加多余的成本。不要重复造轮子，能够共享的工具平台一定要在组织内共享，一致的工具平台对数据的收集和可视化都会带来好处。同时，也要考虑团队优先适配的问题，比如某个团队的自动化测试最佳方案是用工具A，非强调要组织级统一，强制使用工具B，那也是不合适的。万事皆需平衡。

03 组织级质量赋能

根据前面介绍的测试体系图谱，总体来说，组织级质量赋能要着重关注以下几个方面：

1. 树立适应新时代软件质量要求的价值观，营造全员重视质量和价值交付的组织文化

随着科技的持续发展，软件生态日益复杂，新时代软件质量有着更高的要求，软件质量保障也面临更大的挑战，传统的质量保障、软件测试的思路不能完全适合。而是需要改变传统的认知观念，实现多角色深度融合、协同作战、全员为质量负责，重视质量和价值交付。

2. 朝着全流程标准化的方向前行，降低质量成本

全流程标准化是大势所趋，但一步到位也是比较难，分步走的策略才是可行的：

首先，定义组织级统一的流程策略，形成书面标准；
同时，制定实践活动规范，以指导书面标准的落地实施；
通过部分团队实践验证，逐步推广并标准化固定下来，形成组织级标准；
利用工具支撑设置质量门禁等方式确保书面标准能够真正落地，并持续改进和优化策略标准和实践规范。

3. 强化质量基础设施建设，扩大自动化规模，提升研发效能

大规模自动化是实现全流程标准化的有利助手，组织级测试体系需要从测试自动化和流程自动化两个方向加强基础设施建设，两者缺一不可。

4. 人员能力是关键，加强能力建设不容忽视

一方面，制定组织级人才培养机制，针对不同角色建立相应的能力模型和提升路径，形成人才梯度，有计划的实现人员能力建设；同时，鼓励社区建设，营造学习型文化氛围，以促进人员自主学习和提升。

关于测试人员能力提升，可以参考下列文章部分内容：

《数字化转型背景下的测试转型》
《软件测试人员该何去何从》

04 最后

继《基础篇》针对测试人员的基本职责和《进阶篇》针对的全生命周期的质量内建之后，本《高级篇》通过介绍“一个中心，四个方向”的组织级测试体系图谱，以帮助实现组织级质量赋能。

至此，构建测试的体系化思维系列的基础、进阶和高级三个层次的内容均已完成，希望能够提供一个帮助大家思考的测试体系框架。

耐心读完的朋友，如果你有任何反馈，欢迎通过我的个人博客网站「BY林子」给我留言。感谢！

【演讲视频】

【英文版 English Version】

00 Introduction

0.1 Three levels of Systematic Thinking for Testing

Do testing personnel lack systematic thinking?
How to systematically test a newly established product?
How to build a unified testing system at the organizational level?

Many of us have come across countless articles or training courses on testing and quality, which are not lacking in testing practices or technical content, but it is difficult to build our own testing system. Based on similar doubts from many friends and my own team practice and consulting experience over the years, I will discuss the construction of systematic thinking for testing at three different levels: basic, intermediate, and advanced.

0.2 Review of the Basic Level Content

In January 2022, "Building Systematic Thinking for Testing - Intermediate" was published, which introduced the corresponding practical activities and methods for the five basic responsibilities of testing.

Understand and clarify business requirements
Develop strategies and design tests
Implement and execute tests
Defect management and analysis
Quality feedback and risk identification

In March, "Building Systematic Thinking for Testing (Intermediate)" was published, which was about building systematic thinking for testing from a quality perspective.

From testing to quality: Move beyond testing and consider the construction of testing systems from the perspective of quality.
Quality assurance and quality built-in: Quality is inherent in products, and quality built-in is the fundamental guarantee for quality assurance.
In-depth practice of quality built-in: How to implement quality built-in is the most concerned issue, and this article will introduce in-depth practices for reference.

0.3 Overview of Advanced Level Content

This article is the Advanced and will discuss the systematic thinking from building a testing system at the organizational level.

From team to organization: Move beyond product/project teams and consider the systematic construction of testing at the organizational level.
Organizational testing system map: Based on the "one center, four directions" map, explore the construction of organizational testing systems comprehensively.
Organizational quality empowerment: Based on the testing system map, empower the entire organization from an organizational level perspective.

01 From Team to Organization

1.1 Why Move from Team to Organization?

In the "Basic" article, we discussed the basic responsibilities of testing, and in the "Intermediate", we discussed quality built-in, which were mainly within the scope of the team/project. However, to truly build a testing system or systematically consider and carry out quality assurance work, the team's capabilities are still limited. For example, have you encountered familiar scenarios such as:

My organization has many KPIs, and work is driven by the KIPs, but I feel that some work does not bring value.
I am in an independent testing department, and the performance is inconsistent/contradictory with the development team's.
The business department is far away from us, and all development and testing work can only be carried out based on the requirement specification document.
The organization's various review processes are very strict, with high requirements for documents, and a lot of time is spent on writing documents and going through review processes.
The software products developed have dependencies on both upstream and downstream, and a lot of waiting is needed for test data preparation and test environment readiness.
I am not clear about the specific details of the tools and testing techniques used by other teams.
...

The problems in the scenarios above are difficult to solve within a single team. Therefore, the ultimate goal of building a systematic testing system is to look at the construction of testing systems from an organizational perspective.

1.2 What Should Be Considered in An Organizational-level Testing System?

At the organizational level, the focus of the testing system should be on the people, processes, and technologies related to quality assurance. Typically, there are two aspects to consider:

Those closely related to quality and testing, including quality objectives, process specifications, testing strategies, and practice standards;
Those seemingly unrelated to quality and testing, but have a critical impact on the implementation of quality assurance work, such as organizational structure, corporate culture, and the way departments collaborate.

02 Organizational-level Testing System Map

The organizational-level testing system is inevitably much more complex than the team level. To better understand and discuss it, the aspects that need to be considered are organized into the following map:

The map consists of "one center and four directions." To build a complete organizational-level testing system, it is recommended to focus on "one center" and make efforts in the four directions:

One center: core values
Four directions: efficient collaboration, standardized guidance, normative implementation, and automated support.

2.1 Core Values

Core values are the cultural beliefs of an organization that deeply influence the behavior of every member within it. The construction of a testing system should be guided by the core values.

2.1.1 Value-Driven vs. Metric-Driven

The vast majority of organizations use metrics to assess the work of each individual because it is a simple way to measure the quality of the work done. However, this approach is also quite crude because "not everything that can be measured is important, and not everything that is important can be measured."

If we focus too much on metrics, we can become metric-driven, where work is done to meet a certain metric. Whatever is measured, something will surely be obtained. Once a certain metric is established, employees will try their best to meet it in their work, but it becomes difficult to determine whether the truly important work has been completed. This metric-driven approach to work will inevitably ignore the true value behind the work. This is extremely dangerous! In the opening of the book "The Tyranny of Metrics" two classic examples are given: the crime clearance rate of police officers and the success rate of surgeries performed by obstetricians and gynecologists.

Let's take an example of metric-driven software development. The fallacy of "velocity drives everything" mentioned in the article "Velocity Is Not Responsible for It" is an instance of incorrectly using a metric (development velocity) to drive development. It is shown in the following:

I believe that everyone can easily come up with examples of metric-driven software development, so I will not go into detail here.

So, is it not necessary to measure at all? No, appropriate measurement is still necessary. Measurement can serve as a traction to help the team to improve.

However, work cannot be driven by any metric, but should be value-driven. Software development requires delivering software that can bring business value and is of high quality, and work done around this value goal is meaningful. Therefore, software quality assurance and software testing should also be driven by business value. For more on this topic, please refer to the article "Business Value-Driven Testing".

At the organizational level, it is necessary to build a culture that is value-driven, guiding employees to focus more on value and weakening the negative effects of metric-driven work, to prevent falling into "The Tyranny of Metrics".

2.1.2 Encouraging Innovation vs. Blame Culture

The core consequence of performance appraisal is blame culture, which is closely related to KPIs. Whether employees are blamed or improvement measures are taken when performance indicators are not met or accidents occur will directly affect their behavior, attitudes, and problem-solving method.

For example:

In the event of a serious production issue, should the responsible person be blamed and punished, or should the root cause of the issue be analyzed and improvement measures be taken to avoid it in the future?

If the former approach is taken, all relevant personnel will be anxious and it will be difficult to objectively analyze and solve the problem. Instead, certain factors may be concealed, leading to the possibility of similar problems occurring again. Regarding the latter approach, there was a detailed practice of defect analysis in the article "How Defect Analysis Helps Build Quality In" . Here, it is important to emphasize that defect analysis only needs to analyze the objective reasons for the problem, without the need to blame individuals.

A very typical example of non-blame culture is the prime directive followed in agile retrospective meetings:

"Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand."

-- Norm Kerth, Project Retrospectives: A Handbook for Team Review

The other side of blaming is encouraging innovation. Success comes from constant trial and error, and to do things well, one needs to constantly try, experience multiple failures, and summarize experience and lessons learned from each failure to make progress. For example, high-tech companies like Google and Amazon are very encouraging of innovation.

Organizations need to promote this culture of innovation, give employees appropriate space for innovation, encourage everyone to actively try new ideas, and continuously improve the software development process to improve software quality.

2.1.3 Collective Responsibility vs. Independent Separation

For software development teams, "team's responsibility for quality" is almost a household name. However, from an organizational perspective, how should a team be defined? While the principles are understood, how are they implemented at the operational level?

Based on my knowledge of the current situation, there is still a significant gap between departments and different roles within a team. Many people believe that quality is primarily the responsibility of testers (departments/teams/individuals), while business people are responsible for business requirements, developers are responsible for coding, and operations is responsible for the online environment.

I have written several articles on team responsibility for quality before. You are welcome to read them:

From an organizational perspective, it is necessary to clarify quality goals and make all members aware of them to drive collaboration between functional departments based on quality objectives and enhance collective responsibility for quality among all team members.

2.2 Efficient Collaboration

The emphasis of efficient collaboration lies in the organizational structure and the collaborative methods among different parts within the organization. This is a crucial part of building a testing system at the organizational level.

2.2.1 Organizational Structure

"Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." - Melvin E. Conway (Conway's Law)

Software development is a complex social activity that requires sufficient communication and collaboration among business, developers, testers, operations, and other teams. According to Conway's Law, it is necessary to adjust the organizational structure to meet the needs of software development.

However, traditional enterprises typically have thick departmental walls, with business and technology, developers and testers, and operations belonging to different departments, and these walls are difficult to break. This is undoubtedly a major contradiction.

For example, traditional enterprises usually have independent and large testing departments/testing teams, which undertake testing work in independent testing phases. In the new situation that requires continuous testing, how to adjust the testing department to better cooperate efficiently with the development department is a matter of concern and also a problem that needs to be solved in establishing a comprehensive testing system in the organization.

How to solve this problem? There are usually two common ways:

Keep the original independent testing department, with testing personnel managed by the testing department but assigned to the development team to carry out testing work within the development team;
No longer have a testing department, disperse all testing personnel and integrate them into the development center, deeply integrating with the development team.

Which way is better? It's hard to say.

Formally, the second way of completing integration seems to be better, which can achieve deeper integration of developers and testers. However, this also requires high demands for the team, such as doing well in testing shift left, team responsibility for quality, and other aspects, to provide reassuring quality assurance. Therefore, for industries that have very high quality requirements, such as the financial, the first way may be more acceptable. One reason is that breaking through traditional departmental walls is too difficult, and another important aspect is that there is still an independent testing department to "check" the quality, which seems to make everyone more at ease.

As for which form is suitable for your organization, it still needs to be explored by your organization itself.

2.2.2 Team Topologies

"Organizations should be seen as complex, adaptive organisms, rather than as linear, mechanistic systems."

Naomi Stanford, excerpt from "Guide to Organization Design"

The book "Team Topologies: Organizing Business and Technology Teams for Fast Flow" points out the trap of organizational structures: traditional organizational charts are relatively static and do not help us understand the real communication patterns within an organization. In software development, there may be many communication patterns that are not reflected in the organizational chart. These adaptive communication structures that are critical and valuable to software development are emphasized. Therefore, the concept of team topologies is proposed.

The team topologies approach revolves around the effective team structure for enterprise software delivery and brings a new way of thinking. It offers four fundamental team types: Stream-aligned teams, Platform teams, Enabling teams, and Complex subsystem teams, and three core team interaction modes: Collaboration, X-as-a-Service, and Facilitating. These concepts aim to help organizations create efficient and effective teams that are optimized for delivering software products. It emphasizes focusing more on actual communication paths outside of the static organizational structure (team structure) and supplementing the dynamic and perceptual capabilities that are lacking in traditional organizational design for technical organizations.

Looking back at the issue of how to organize the testing department mentioned earlier, based on the team topologies concept, we may think of different solutions. Strengthening the interaction between testing and various departments/roles is the key. We need to build a flexible and adaptable team topology structure based on product development needs. The specific organizational form may not be the most important.

2.2.3 Communication and collaboration

Team Topologies emphasizes the importance of communication structures within an organization, but just building teams with the right communication patterns doesn't guarantee the desired communication needs will be met.

For example, you may have experienced or heard about situations such as:

Only "upper-level" leaders know some business goals and quality objectives and are not transparent to the entire organization or team.
Some companies have business and development teams sitting together, but communication still happens through requirement documents, just as before.
In some teams, testers and developers belong to the same team, but testers are not involved in technical discussions, developers don't inform testers of technical updates, and developers don't participate in the formulation of testing strategies and plans.
Different small teams of the same product or project operate independently, without knowledge or information sharing, leading to repeated wheel reinvention.

There may be many similar situations. These are seemingly fine on the surface but lack effective communication and collaboration.

Regarding communication, it's important to value the content of communication, making information transparent throughout the organization or team, especially regarding vision, goals, and risk information, which can enhance members' sense of mission and subjective initiative. In my article "Guiding Principles of Agile Testing", I introduced the characteristics of efficient and collaborative teams, and in "Beware of 'Mushroom Farming' in Teams", I shared the importance of information sharing by examples.

On the other hand, it's also important to pay attention to the forms of communication, as different forms of communication have vastly different effects. Typically, face-to-face communication is the most effective way, as shown in the figure below:

Collaboration is built on the premise of adequate communication. Once communication is in place, collaboration will be smoother.

2.3 Standardized Guidance

I have previously written an article titled "Quality Assurance Empowerment for Agile Teams", in which I investigated the software development and delivery processes of large companies such as Google, Amazon, and Facebook, and summarized that team quality assurance empowerment needs to achieve standardized processes throughout the entire process.

By guiding the standardization of work methods, teams can better ensure quality at every stage and take responsibility for quality as a team. It is important to note that standardization should not be limited to written forms only, as described in "Factual Standards and Written Standards" in the article "Quality Assurance Empowerment for Agile Teams)".

2.3.1 Standardization of Development Processes

Testing and development are inseparable, and testing strategies are influenced by different development processes.

Whether it is a traditional waterfall development process, an iterative agile development process, or a combination of the two, the corresponding process strategies need to be standardized, with clear definitions of the activities in each stage and the responsibilities of different team members at the organizational level.

With standardized guidance for the development process, teams should conduct detailed analyses of bottlenecks in any stage and make adjustments and improvements if the process no longer fits the current team's situation.

2.3.2 Standardization of Testing Strategy

For different development processes, there needs to be a standardized testing strategy at the organizational level. The organizational-level testing strategy serves as a unified guiding direction, and different product lines and project teams also need space for autonomous adjustment to better customize and adapt their strategies.

In addition to providing space for teams to adjust their strategies, it should also be emphasized that strategies are not set in stone and need to be adapted and adjusted based on quality risks and team status during different periods. It should be evolutionary.

Regarding testing strategy, it is strongly recommended to refer to "One-Page Test Strategy" and customize it based on the characteristics of the organization.

2.3.3 Standardization of Quality Objectives and Measurement Strategies

Clear quality objectives can better drive the construction of the organizational-level testing system. Organizational-level quality objectives should not only be quality requirements for pure system use but also closely related to business objectives and driven by business value. Regarding the relationship between business value and testing, you can refer to the following articles:

When it comes to quality objectives, corresponding measurement indicators and strategies are necessary. Organizational-level measurement indicators should be defined for quality objectives and measured through a unified measurement strategy. In terms of measurement, it is especially important to not focus on a single indicator but rather to use a combination of indicators for measurement. Indicators should not be used as a standard for horizontal comparison between product lines/teams but rather as a benchmark to guide product lines/teams' continuous improvement. I strongly agree with my Thoughtworks colleague Ben WU's statement in his article "Measurement is to Identify the Biggest Bottleneck in Value Stream":

"In agile IT development and delivery, measurement plays a role in identifying the biggest bottleneck in the value stream, in order to identify the biggest bottleneck (and direction error) in end-to-end value flow among the dimensions of 'accurate value, fast flow, and good quality,' and make it the next improvement point for improvement, in order to maximize the effectiveness of improvement."

Regarding quality measurement, my Thoughtworks colleague YU Xiaonan has shared very detailed insights from both qualitative and quantitative perspectives and has written a series of related articles. You can refer to her "Quality Measurement Collection" for more information.

2.4 Normative Implementation

Under the guidance of standardized strategy, it is a challenging task to ensure that each practical activity is well-implemented. Some practices may be implemented incorrectly from the beginning, or they may start off correctly but eventually become outdated.

The implementation plan for standardization is a guarantee for the landing and implementation of standardized strategies. Organizational levels need to consider the effective implementation of each practical activity from the following aspects: defining practice standards, precipitating new excellent practices, customizing maturity models, and conducting regular governance and continuous improvement.

2.4.1 Define Practice Standards

Clear definitions should be provided for each practical activity, including the practice's objectives, applicable conditions, participating roles, inputs and outputs, and it is best to introduce the entire practice activity in detail using examples. Canvas can be used to introduce the above items more clearly and intuitively.

The defined practice standards are the reference for the team's implementation and landing. During the team's practice process, the team is encouraged to improve each practical activity based on their experience, achieving continuous optimization.

2.4.2 Precipitate New Excellent Practices

During the practice process, the team may also discover new practical activities. It is recommended to share these new practical activities with more teams, recommend them for trial, and once they are recognized by multiple teams, standardize them and store them in the organizational practice library.

At the same time, practices that are no longer applicable should also be eliminated and removed.

2.4.3 Customized maturity model

Based on the practice standards, a maturity model can be developed to evaluate the implementation status of each stage of the team's practices and formulate corresponding improvement measures to help the team grow continuously. The organization needs to develop a guiding maturity model for each team to customize and guide their improvement. For more information, please refer to my article on "Testing Assessment".

It is important to emphasize that the purpose of the maturity model is not for evaluation but for guiding improvement. If maturity is to be associated with team performance, it must be approached with caution, or else it may fall into The Tyranny of Metrics.

2.4.4 Regular governance and continuous improvement

Just having a maturity model is not enough. Regular governance is important to guide improvement. At the organizational level, a quality governance strategy should be defined and applied to each team. Based on the maturity model, teams should periodically review the implementation status of their practices, assess the current situation, and develop new improvement measures based on the review and evaluation results, combined with current quality goals.

For example, teams can be encouraged to conduct regular reviews and governance based on iteration and release cycles. Also, different improvement measures may require different cycles for governance and improvement.

In summary, the organization should have a sense of regular governance and continuous improvement, combined with the maturity model, to encourage teams to achieve continuous improvement.

2.5 Automated Support

A strong infrastructure can provide automated support for the implementation of quality practices, simultaneously improving quality and efficiency.

From the perspective of building a testing system, organizations need to consider automation support in several areas:

Process management: The software delivery process throughout the entire lifecycle usually requires pipeline support.
Testing framework: Various automation testing frameworks and their association with continuous integration pipelines.
Data analysis: Visualization of various data and analysis results, including log information and metric data.
Monitoring and warning: Real-time monitoring of team and product status, intelligent prediction, and warning of quality risks and severe defects.

Whenever efficiency is mentioned, various efficiency support platforms and tools come to mind. However, given the enthusiasm for tool platforms, it is essential to remind of a few points:

Emphasize the use of tool platforms, but personnel lack the corresponding empowerment, are not clear about the principles behind the tool platforms, and cannot use them efficiently. Both platform vendors and purchasing enterprises have talked to me about this phenomenon, indicating that it is not an isolated case. Even if the tool platform is excellent, it can only achieve twice the result with half the effort if used correctly; otherwise, it will become a burden.
Standards and specifications remain on paper and have not been implemented. This is what was mentioned earlier in "Quality Empowerment of Agile Teams", the section of "'real standards' and 'written standards'", and tool platforms can help turn written standards into real standards.
Different teams use different tool frameworks, and the organization does not have a unified tool platform. This will not be conducive to collaboration and integration between different teams, will not be conducive to collecting organization-level data, and will increase unnecessary costs. Do not reinvent the wheel. Tool platforms that can be shared within the organization should be shared, and a consistent tool platform will bring benefits to data collection and visualization. At the same time, the issue of team priority adaptation must also be considered. For example, it is not appropriate to force the use of tool B if team A's best solution for automated testing is tool A, even if there is an emphasis on organizational-level unity. All things require balance.

03 Quality Enablement at the Organizational Level

Based on the test system map introduced earlier, overall, quality enablement at the organizational level should focus on the following aspects:

3.1 Establishing a value system that adapts to the new era's software quality requirements and fostering an organizational culture where everyone values quality and value delivery

With the continuous development of technology, the software ecosystem is becoming increasingly complex, and new era software quality has higher requirements. Traditional quality assurance and software testing approaches may not fully adapt to the changing landscape. Therefore, it is necessary to change traditional perceptions, achieve deep integration of multiple roles, collaborate, and take responsibility for quality and value delivery.

3.2 Moving towards standardized processes throughout the entire process and reducing quality costs

Standardizing processes throughout the entire process is the trend, but it is also challenging to accomplish in one step. A step-by-step approach is more feasible:

Firstly, define organization-wide process strategies and form written standards;
At the same time, develop practice activity specifications to guide the implementation of written standards;
Gradually promote and standardize by verifying through practical tests of some teams to form organizational standards;
Utilize tool support to set up quality gates and other methods to ensure that written standards can truly be implemented, and continuously improve and optimize strategic standards and practice specifications.

3.3 Strengthen quality infrastructure construction, expand automation scale, and improve R&D efficiency

Large-scale automation is a helpful way for achieving standardized processes throughout the organization. The organizational-level testing system needs to strengthen infrastructure construction in both testing automation and process automation, as both are essential.

3.4 The capability of quality personnel is crucial, and enhancing capability building cannot be ignored.

On the one hand, it is essential to establish organizational-level talent development mechanisms, to establish corresponding capability models and development paths for different roles, to form a talent gradient, and to implement personnel capability development in a planned manner. At the same time, community building should be encouraged, and a learning culture should be created to promote personnel's autonomous learning and development.

Regarding the improvement of quality personnel capabilities, you can refer to the following articles:

04 Conclusion

Following the Basic level that focused on the basic responsibilities of testers and the Intermediate level that focused on quality built-in throughout the entire lifecycle, this Advanced level introduces the "one center, four directions" organizational test system map to help enable organization-level quality.

With this, the series of systematic thinking for testing at the basic, intermediate, and advanced levels are all complete, hoping to provide a testing system framework that can help everyone think about testing.