You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

178 lines
3.9 KiB

pipeline: Directed graph execution model Our current pipelinelib based jobs require repos to conform to a number of rigid conventions: assume the repo contains source for only a single application, build "test" variant, run "test" variant, build "production" variant, helm deploy/test, publish under a single tag name. These jobs also assume all of these operations need to be performed linearly. While this design was sufficient for our very first use cases, its convention based design it already proving prohibitively inflexible. For example, teams maintaining repos that contain multiple interrelated applications cannot build and test these applications as independent images; Teams wanting to execute multiple test suites would have to wrap them in a single entrypoint and implement their own concurrency should they need it; Etc. Instead of Release Engineering maintaining a new specialized pipeline job for each team that performs only slightly different permutations of the same operations (resulting in duplication of job definitions and a large maintenance burden), we can instead establish a configuration format and interface by which teams provide their own pipeline compositions. This initial commit in a series of pipeline related commits implements two fundamental components to support a CI/CD pipeline that can execute any number of user-defined variant build/test/publish/deploy stages and steps in a safely concurrent model: a directed-graph based execution model, and name bindings for stage outputs. The former provides the model for composing stage execution, and the latter provides a decoupled system for defining what outputs each subsequent stage operates upon. First, an `ExecutionGraph` class that can represent a directed acyclic graph given a number of linearly defined arcs (aka branches/edges). This component will allow users to provide the overall execution flow as separate linear processes but allow parallel branches of the execution graph to be scheduled concurrently. Example: /* To represent a graph with separate parallel branches like: * * a x * ⇘ ⇙ * b * ⇙ ⇘ * y c * ⇘ ⇙ * z * * One only needs to provides each linear execution arc */ def graph = new ExecutionGraph([["a", "b", "c", "z"], ["x", "b", "y", "z"]]) /* The ExecutionGraph can solve how those arcs intersect and how the * nodes can be scheduled with a degree of concurrency that Jenkins * allows. */ graph.stack() // => [["a", "x"], ["b"], ["y", "c"], ["z"]] Second, a set of context classes for managing immutable global and local name/value bindings between nodes in the graph. Effectively this will provide a way for pipeline stages to safely and deterministically consume inputs from previous stages along the same branch, and to provide their own outputs for subsequent stages to consume. For example, one stage called "build" that builds a container image will save the image ID in a predetermined local binding called `.imageID` and a subsequent "publish" stage configured by the user can reference that image by `${build.imageID}`. Once a value is bound to a name, that name cannot be reused; bindings are immutable. Node contexts are only allowed to access namespaces for nodes that precede them in same branch of the graph, ensuring deterministic behavior during parallel graph branch execution. See unit tests for `ExecutionContext` for details on expected behavior. Put together, these two data structures can constitute an execution "stack" of sorts that can be safely mapped to Jenkins Pipeline stages, and make use of parallel execution for graph branches. Specifically, the `ExecutionGraph.stack()` method is implemented to yield each set of independent stack "frames" in topological sort order which can safely be scheduled to run in parallel. Bug: T210267 Change-Id: Ic5d01bf54c703eaf14434a36f1e2b3e276b48b6f
5 years ago
  1. import groovy.mock.interceptor.MockFor
  2. import static groovy.test.GroovyAssert.*
  3. import groovy.util.GroovyTestCase
  4. import org.wikimedia.integration.ExecutionGraph
  5. class ExecutionGraphTest extends GroovyTestCase {
  6. void testAncestorsOf() {
  7. def graph = new ExecutionGraph([
  8. ["a", "b", "c", "d", "e", "f"],
  9. ["x", "d", "y", "f"],
  10. ])
  11. assert graph.ancestorsOf("c") == ["b", "a"] as Set
  12. assert graph.ancestorsOf("d") == ["c", "b", "a", "x"] as Set
  13. assert graph.ancestorsOf("f") == ["e", "d", "c", "b", "a", "x", "y"] as Set
  14. }
  15. void testLeaves() {
  16. def graph = new ExecutionGraph([
  17. ["a", "b", "c", "d"],
  18. ["f", "b", "g"],
  19. ])
  20. assert graph.leaves() == ["g", "d"] as Set
  21. }
  22. void testInDegreeOf() {
  23. def graph = new ExecutionGraph([
  24. ["a", "b", "c"],
  25. ["d", "b", "e"],
  26. ["f", "b"],
  27. ])
  28. assert graph.inDegreeOf("a") == 0
  29. assert graph.inDegreeOf("d") == 0
  30. assert graph.inDegreeOf("f") == 0
  31. assert graph.inDegreeOf("b") == 3
  32. assert graph.inDegreeOf("e") == 1
  33. assert graph.inDegreeOf("c") == 1
  34. }
  35. void testInTo() {
  36. def graph = new ExecutionGraph([
  37. ["a", "b", "c"],
  38. ["d", "b", "e"],
  39. ["f", "b"],
  40. ])
  41. assert graph.inTo("a") == [] as Set
  42. assert graph.inTo("d") == [] as Set
  43. assert graph.inTo("f") == [] as Set
  44. assert graph.inTo("b") == ["a", "d", "f"] as Set
  45. assert graph.inTo("c") == ["b"] as Set
  46. assert graph.inTo("e") == ["b"] as Set
  47. }
  48. void testNodes() {
  49. def graph = new ExecutionGraph([
  50. ["a", "b", "c"],
  51. ["x", "b", "y"],
  52. ["z"],
  53. ])
  54. assert graph.nodes() == ["a", "b", "c", "x", "y", "z"] as Set
  55. }
  56. void testOr() {
  57. def graph1 = new ExecutionGraph([
  58. ["x", "y", "z"],
  59. ])
  60. def graph2 = new ExecutionGraph([
  61. ["a", "b", "c"],
  62. ["d", "b", "e"],
  63. ])
  64. assert (graph1 | graph2) == new ExecutionGraph([
  65. ["x", "y", "z"],
  66. ["a", "b", "c"],
  67. ["d", "b", "e"],
  68. ])
  69. }
  70. void testOutDegreeOf() {
  71. def graph = new ExecutionGraph([
  72. ["a", "b", "c"],
  73. ["d", "b", "e"],
  74. ["f", "b"],
  75. ])
  76. assert graph.outDegreeOf("a") == 1
  77. assert graph.outDegreeOf("d") == 1
  78. assert graph.outDegreeOf("f") == 1
  79. assert graph.outDegreeOf("b") == 2
  80. assert graph.outDegreeOf("e") == 0
  81. assert graph.outDegreeOf("c") == 0
  82. }
  83. void testOutOf() {
  84. def graph = new ExecutionGraph([
  85. ["a", "b", "c"],
  86. ["d", "b", "e"],
  87. ["f", "b"],
  88. ])
  89. assert graph.outOf("a") == ["b"] as Set
  90. assert graph.outOf("d") == ["b"] as Set
  91. assert graph.outOf("f") == ["b"] as Set
  92. assert graph.outOf("b") == ["c", "e"] as Set
  93. assert graph.outOf("c") == [] as Set
  94. assert graph.outOf("e") == [] as Set
  95. }
  96. void testPlus() {
  97. def graph1 = new ExecutionGraph([
  98. ["x", "y", "z"],
  99. ])
  100. def graph2 = new ExecutionGraph([
  101. ["a", "b", "c"],
  102. ["d", "b", "e"],
  103. ])
  104. assert (graph1 + graph2) == new ExecutionGraph([
  105. ["x", "y", "z", "a", "b", "c"],
  106. ["x", "y", "z", "d", "b", "e"],
  107. ])
  108. }
  109. void testRoots() {
  110. def graph = new ExecutionGraph([
  111. ["a", "b", "c", "d", "e"],
  112. ["f", "b", "g", "e"],
  113. ])
  114. assert graph.roots() == ["a", "f"] as Set
  115. }
  116. void testRootsOfIsolates() {
  117. def graph = new ExecutionGraph([["a"], ["z"]])
  118. assert graph.roots() == ["a", "z"] as Set
  119. }
  120. void testStack() {
  121. def graph = new ExecutionGraph([
  122. ["a", "b", "c", "d", "e", "f"],
  123. ["x", "d", "y", "f"],
  124. ])
  125. assert graph.stack() == [
  126. ["a", "x"],
  127. ["b"],
  128. ["c"],
  129. ["d"],
  130. ["e", "y"],
  131. ["f"],
  132. ]
  133. }
  134. void testStack_cycleDetection() {
  135. def graph = new ExecutionGraph([
  136. ["a", "b", "c"],
  137. ["x", "b", "y", "a"],
  138. ])
  139. shouldFail(Exception) {
  140. graph.stack()
  141. }
  142. }
  143. void testToString() {
  144. def graph = new ExecutionGraph([["a", "b", "c"], ["x"]])
  145. assert graph.toString() == "digraph { a -> b; b -> c; x }"
  146. }
  147. }