You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

177 lines
4.3 KiB

pipeline: Directed graph execution model Our current pipelinelib based jobs require repos to conform to a number of rigid conventions: assume the repo contains source for only a single application, build "test" variant, run "test" variant, build "production" variant, helm deploy/test, publish under a single tag name. These jobs also assume all of these operations need to be performed linearly. While this design was sufficient for our very first use cases, its convention based design it already proving prohibitively inflexible. For example, teams maintaining repos that contain multiple interrelated applications cannot build and test these applications as independent images; Teams wanting to execute multiple test suites would have to wrap them in a single entrypoint and implement their own concurrency should they need it; Etc. Instead of Release Engineering maintaining a new specialized pipeline job for each team that performs only slightly different permutations of the same operations (resulting in duplication of job definitions and a large maintenance burden), we can instead establish a configuration format and interface by which teams provide their own pipeline compositions. This initial commit in a series of pipeline related commits implements two fundamental components to support a CI/CD pipeline that can execute any number of user-defined variant build/test/publish/deploy stages and steps in a safely concurrent model: a directed-graph based execution model, and name bindings for stage outputs. The former provides the model for composing stage execution, and the latter provides a decoupled system for defining what outputs each subsequent stage operates upon. First, an `ExecutionGraph` class that can represent a directed acyclic graph given a number of linearly defined arcs (aka branches/edges). This component will allow users to provide the overall execution flow as separate linear processes but allow parallel branches of the execution graph to be scheduled concurrently. Example: /* To represent a graph with separate parallel branches like: * * a x * ⇘ ⇙ * b * ⇙ ⇘ * y c * ⇘ ⇙ * z * * One only needs to provides each linear execution arc */ def graph = new ExecutionGraph([["a", "b", "c", "z"], ["x", "b", "y", "z"]]) /* The ExecutionGraph can solve how those arcs intersect and how the * nodes can be scheduled with a degree of concurrency that Jenkins * allows. */ graph.stack() // => [["a", "x"], ["b"], ["y", "c"], ["z"]] Second, a set of context classes for managing immutable global and local name/value bindings between nodes in the graph. Effectively this will provide a way for pipeline stages to safely and deterministically consume inputs from previous stages along the same branch, and to provide their own outputs for subsequent stages to consume. For example, one stage called "build" that builds a container image will save the image ID in a predetermined local binding called `.imageID` and a subsequent "publish" stage configured by the user can reference that image by `${build.imageID}`. Once a value is bound to a name, that name cannot be reused; bindings are immutable. Node contexts are only allowed to access namespaces for nodes that precede them in same branch of the graph, ensuring deterministic behavior during parallel graph branch execution. See unit tests for `ExecutionContext` for details on expected behavior. Put together, these two data structures can constitute an execution "stack" of sorts that can be safely mapped to Jenkins Pipeline stages, and make use of parallel execution for graph branches. Specifically, the `ExecutionGraph.stack()` method is implemented to yield each set of independent stack "frames" in topological sort order which can safely be scheduled to run in parallel. Bug: T210267 Change-Id: Ic5d01bf54c703eaf14434a36f1e2b3e276b48b6f
5 years ago
pipeline: Directed graph execution model Our current pipelinelib based jobs require repos to conform to a number of rigid conventions: assume the repo contains source for only a single application, build "test" variant, run "test" variant, build "production" variant, helm deploy/test, publish under a single tag name. These jobs also assume all of these operations need to be performed linearly. While this design was sufficient for our very first use cases, its convention based design it already proving prohibitively inflexible. For example, teams maintaining repos that contain multiple interrelated applications cannot build and test these applications as independent images; Teams wanting to execute multiple test suites would have to wrap them in a single entrypoint and implement their own concurrency should they need it; Etc. Instead of Release Engineering maintaining a new specialized pipeline job for each team that performs only slightly different permutations of the same operations (resulting in duplication of job definitions and a large maintenance burden), we can instead establish a configuration format and interface by which teams provide their own pipeline compositions. This initial commit in a series of pipeline related commits implements two fundamental components to support a CI/CD pipeline that can execute any number of user-defined variant build/test/publish/deploy stages and steps in a safely concurrent model: a directed-graph based execution model, and name bindings for stage outputs. The former provides the model for composing stage execution, and the latter provides a decoupled system for defining what outputs each subsequent stage operates upon. First, an `ExecutionGraph` class that can represent a directed acyclic graph given a number of linearly defined arcs (aka branches/edges). This component will allow users to provide the overall execution flow as separate linear processes but allow parallel branches of the execution graph to be scheduled concurrently. Example: /* To represent a graph with separate parallel branches like: * * a x * ⇘ ⇙ * b * ⇙ ⇘ * y c * ⇘ ⇙ * z * * One only needs to provides each linear execution arc */ def graph = new ExecutionGraph([["a", "b", "c", "z"], ["x", "b", "y", "z"]]) /* The ExecutionGraph can solve how those arcs intersect and how the * nodes can be scheduled with a degree of concurrency that Jenkins * allows. */ graph.stack() // => [["a", "x"], ["b"], ["y", "c"], ["z"]] Second, a set of context classes for managing immutable global and local name/value bindings between nodes in the graph. Effectively this will provide a way for pipeline stages to safely and deterministically consume inputs from previous stages along the same branch, and to provide their own outputs for subsequent stages to consume. For example, one stage called "build" that builds a container image will save the image ID in a predetermined local binding called `.imageID` and a subsequent "publish" stage configured by the user can reference that image by `${build.imageID}`. Once a value is bound to a name, that name cannot be reused; bindings are immutable. Node contexts are only allowed to access namespaces for nodes that precede them in same branch of the graph, ensuring deterministic behavior during parallel graph branch execution. See unit tests for `ExecutionContext` for details on expected behavior. Put together, these two data structures can constitute an execution "stack" of sorts that can be safely mapped to Jenkins Pipeline stages, and make use of parallel execution for graph branches. Specifically, the `ExecutionGraph.stack()` method is implemented to yield each set of independent stack "frames" in topological sort order which can safely be scheduled to run in parallel. Bug: T210267 Change-Id: Ic5d01bf54c703eaf14434a36f1e2b3e276b48b6f
5 years ago
  1. import groovy.mock.interceptor.MockFor
  2. import static groovy.test.GroovyAssert.*
  3. import groovy.util.GroovyTestCase
  4. import org.wikimedia.integration.ExecutionGraph
  5. import org.wikimedia.integration.ExecutionContext
  6. class ExecutionContextTest extends GroovyTestCase {
  7. void testNodeContextBindings() {
  8. /*
  9. * a
  10. *
  11. * b
  12. *
  13. * c x
  14. *
  15. * d
  16. *
  17. * y e
  18. *
  19. * f
  20. */
  21. def context = new ExecutionContext(new ExecutionGraph([
  22. ["a", "b", "c", "d", "e", "f"],
  23. ["x", "d", "y", "f"],
  24. ]))
  25. def cContext = context.ofNode("c")
  26. def eContext = context.ofNode("e")
  27. context.ofNode("a").bind("foo", "afoo")
  28. context.ofNode("a").bind("bar", "abar")
  29. context.ofNode("b").bind("foo", "bfoo")
  30. context.ofNode("x").bind("bar", "xbar")
  31. assert cContext.binding("a", "foo") == "afoo"
  32. assert cContext.binding("a", "bar") == "abar"
  33. assert cContext.binding("b", "foo") == "bfoo"
  34. shouldFail(ExecutionContext.AncestorNotFoundException) {
  35. cContext.binding("x", "bar")
  36. }
  37. shouldFail(ExecutionContext.NameNotFoundException) {
  38. cContext.binding("baz")
  39. }
  40. assert eContext.binding("a", "foo") == "afoo"
  41. assert eContext.binding("a", "bar") == "abar"
  42. assert eContext.binding("b", "foo") == "bfoo"
  43. assert eContext.binding("x", "bar") == "xbar"
  44. }
  45. void testNodeContextBindings_immutable() {
  46. /*
  47. * a
  48. *
  49. * b
  50. *
  51. * c
  52. */
  53. def context = new ExecutionContext(new ExecutionGraph([
  54. ["a", "b", "c"],
  55. ]))
  56. context.ofNode("b").bind("foo", "bfoo")
  57. shouldFail(ExecutionContext.NameAlreadyBoundException) {
  58. context.ofNode("b").bind("foo", "newfoo")
  59. }
  60. assert context.ofNode("b").binding("foo") == "bfoo"
  61. }
  62. void testStringInterpolation() {
  63. /*
  64. * a
  65. *
  66. * b
  67. *
  68. * c x
  69. *
  70. * d
  71. *
  72. * y e
  73. *
  74. * f
  75. */
  76. def context = new ExecutionContext(new ExecutionGraph([
  77. ["a", "b", "c", "d", "e", "f"],
  78. ["x", "d", "y", "f"],
  79. ]))
  80. context.ofNode("a").bind("foo", "afoo")
  81. context.ofNode("a").bind("bar", "abar")
  82. context.ofNode("b").bind("foo", "bfoo")
  83. context.ofNode("x").bind("bar", "xbar")
  84. assert (context.ofNode("b") % 'w-t-${a.foo} ${.foo}') == "w-t-afoo bfoo"
  85. assert (context.ofNode("c") % 'w-t-${a.foo} ${b.foo}') == "w-t-afoo bfoo"
  86. assert (context.ofNode("c") % 'w-t-${a.bar}') == "w-t-abar"
  87. shouldFail(ExecutionContext.AncestorNotFoundException) {
  88. context.ofNode("x") % 'w-t-${b.bar}'
  89. }
  90. shouldFail(ExecutionContext.NameNotFoundException) {
  91. context.ofNode("b") % 'w-t-${.bar}'
  92. }
  93. }
  94. void testStringInterpolation_selfScope() {
  95. /*
  96. * a
  97. *
  98. * b
  99. *
  100. * c
  101. */
  102. def context = new ExecutionContext(new ExecutionGraph([
  103. ["a", "b", "c"],
  104. ]))
  105. context.ofNode("c").bind("foo", "cfoo")
  106. assert (context.ofNode("c") % 'w-t-${.foo}') == "w-t-cfoo"
  107. }
  108. void testNodeContextGetAll() {
  109. def context = new ExecutionContext(new ExecutionGraph([
  110. ["a", "b", "c", "z"],
  111. ["x", "b", "y", "z"],
  112. ]))
  113. context.ofNode("a").bind("foo", "afoo")
  114. context.ofNode("c").bind("foo", "cfoo")
  115. context.ofNode("y").bind("foo", "yfoo")
  116. assert context.ofNode("z").getAll("foo") == [
  117. a: "afoo",
  118. c: "cfoo",
  119. y: "yfoo",
  120. ]
  121. }
  122. void testNodeContextGetAt() {
  123. def context = new ExecutionContext(new ExecutionGraph([
  124. ["a", "b", "c"],
  125. ]))
  126. context.ofNode("a").bind("foo", "afoo")
  127. context.ofNode("c").bind("foo", "cfoo")
  128. assert context.ofNode("c")["a.foo"] == "afoo"
  129. assert context.ofNode("c")["foo"] == "cfoo"
  130. }
  131. void testNodeContextGetAt_selfScope() {
  132. /*
  133. * a
  134. *
  135. * b
  136. *
  137. * c
  138. */
  139. def context = new ExecutionContext(new ExecutionGraph([
  140. ["a", "b", "c"],
  141. ]))
  142. def bContext = context.ofNode("b")
  143. bContext["foo"] = "bfoo"
  144. assert bContext[".foo"] == "bfoo"
  145. }
  146. }