You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

202 lines
4.6 KiB

pipeline: Builder and stage implementation Provides `PipelineBuilder` for reading `.pipeline/config.yaml` and mapping user-defined pipelines, stages, and execution graphs to actual Jenkins Pipeline stage definitions. Provides `Pipeline` class that constructs a "stack" of `PipelineStage` objects from the user-provided configs, each with its own `NodeContext` for binding output values to names and consuming bound values from previous stages. Provides `PipelineStage` that contains core stage step implementations based on the existing `service-pipeline` JJB job definition in `integration/config`. A closure is returned by each stage for passing off to Jenkins Pipeline stage definitions by the builder. Steps have a fixed order within a given stage: build, run, publish, deploy, exports. This allows for concise definition of a stage that performs multiple steps, and deterministic behavior of default configuration that references locally bound output values (e.g. the default configuration of `image:` for an `publish: { type: image }` publish entry is `${.imageID}`, referencing the image built in the current stage's `build` step.) If the user needs to change ordering, they can simply break the stage out into multiple stages. See the `Pipeline` class for currently supported configuration. Note that the aforementioned context system allows for user's to make use of the same value bindings that step implementations use internally. They can also use the `exports` configuration field to bind new values. To illustrate the minimally required configuration, the following would approximate the current `service-pipeline-test-and-publish` JJB job for a project named "foo". pipelines: foo: directory: src/foo stages: - name: test # builds/runs "test" variant - name: candidate build: production publish: image: true deploy: # currently only the "ci" cluster chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true And to illustrate how the "candidate" stage in this example could be expressed as multiple stages using references to the output names that steps bind/export: pipelines: foo: directory: src/foo stages: - name: tested - name: built build: production - name: published publish: image: id: '${built.imageID}' exports: image: '${.imageFullName}:${.imageTag}' - name: staged deploy: image: '${published.image}' chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true Bug: T210267 Change-Id: I5a41d0d33ed7e9174db6178ab7921f5143296c75
5 years ago
pipeline: Builder and stage implementation Provides `PipelineBuilder` for reading `.pipeline/config.yaml` and mapping user-defined pipelines, stages, and execution graphs to actual Jenkins Pipeline stage definitions. Provides `Pipeline` class that constructs a "stack" of `PipelineStage` objects from the user-provided configs, each with its own `NodeContext` for binding output values to names and consuming bound values from previous stages. Provides `PipelineStage` that contains core stage step implementations based on the existing `service-pipeline` JJB job definition in `integration/config`. A closure is returned by each stage for passing off to Jenkins Pipeline stage definitions by the builder. Steps have a fixed order within a given stage: build, run, publish, deploy, exports. This allows for concise definition of a stage that performs multiple steps, and deterministic behavior of default configuration that references locally bound output values (e.g. the default configuration of `image:` for an `publish: { type: image }` publish entry is `${.imageID}`, referencing the image built in the current stage's `build` step.) If the user needs to change ordering, they can simply break the stage out into multiple stages. See the `Pipeline` class for currently supported configuration. Note that the aforementioned context system allows for user's to make use of the same value bindings that step implementations use internally. They can also use the `exports` configuration field to bind new values. To illustrate the minimally required configuration, the following would approximate the current `service-pipeline-test-and-publish` JJB job for a project named "foo". pipelines: foo: directory: src/foo stages: - name: test # builds/runs "test" variant - name: candidate build: production publish: image: true deploy: # currently only the "ci" cluster chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true And to illustrate how the "candidate" stage in this example could be expressed as multiple stages using references to the output names that steps bind/export: pipelines: foo: directory: src/foo stages: - name: tested - name: built build: production - name: published publish: image: id: '${built.imageID}' exports: image: '${.imageFullName}:${.imageTag}' - name: staged deploy: image: '${published.image}' chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true Bug: T210267 Change-Id: I5a41d0d33ed7e9174db6178ab7921f5143296c75
5 years ago
pipeline: Builder and stage implementation Provides `PipelineBuilder` for reading `.pipeline/config.yaml` and mapping user-defined pipelines, stages, and execution graphs to actual Jenkins Pipeline stage definitions. Provides `Pipeline` class that constructs a "stack" of `PipelineStage` objects from the user-provided configs, each with its own `NodeContext` for binding output values to names and consuming bound values from previous stages. Provides `PipelineStage` that contains core stage step implementations based on the existing `service-pipeline` JJB job definition in `integration/config`. A closure is returned by each stage for passing off to Jenkins Pipeline stage definitions by the builder. Steps have a fixed order within a given stage: build, run, publish, deploy, exports. This allows for concise definition of a stage that performs multiple steps, and deterministic behavior of default configuration that references locally bound output values (e.g. the default configuration of `image:` for an `publish: { type: image }` publish entry is `${.imageID}`, referencing the image built in the current stage's `build` step.) If the user needs to change ordering, they can simply break the stage out into multiple stages. See the `Pipeline` class for currently supported configuration. Note that the aforementioned context system allows for user's to make use of the same value bindings that step implementations use internally. They can also use the `exports` configuration field to bind new values. To illustrate the minimally required configuration, the following would approximate the current `service-pipeline-test-and-publish` JJB job for a project named "foo". pipelines: foo: directory: src/foo stages: - name: test # builds/runs "test" variant - name: candidate build: production publish: image: true deploy: # currently only the "ci" cluster chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true And to illustrate how the "candidate" stage in this example could be expressed as multiple stages using references to the output names that steps bind/export: pipelines: foo: directory: src/foo stages: - name: tested - name: built build: production - name: published publish: image: id: '${built.imageID}' exports: image: '${.imageFullName}:${.imageTag}' - name: staged deploy: image: '${published.image}' chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true Bug: T210267 Change-Id: I5a41d0d33ed7e9174db6178ab7921f5143296c75
5 years ago
pipeline: Builder and stage implementation Provides `PipelineBuilder` for reading `.pipeline/config.yaml` and mapping user-defined pipelines, stages, and execution graphs to actual Jenkins Pipeline stage definitions. Provides `Pipeline` class that constructs a "stack" of `PipelineStage` objects from the user-provided configs, each with its own `NodeContext` for binding output values to names and consuming bound values from previous stages. Provides `PipelineStage` that contains core stage step implementations based on the existing `service-pipeline` JJB job definition in `integration/config`. A closure is returned by each stage for passing off to Jenkins Pipeline stage definitions by the builder. Steps have a fixed order within a given stage: build, run, publish, deploy, exports. This allows for concise definition of a stage that performs multiple steps, and deterministic behavior of default configuration that references locally bound output values (e.g. the default configuration of `image:` for an `publish: { type: image }` publish entry is `${.imageID}`, referencing the image built in the current stage's `build` step.) If the user needs to change ordering, they can simply break the stage out into multiple stages. See the `Pipeline` class for currently supported configuration. Note that the aforementioned context system allows for user's to make use of the same value bindings that step implementations use internally. They can also use the `exports` configuration field to bind new values. To illustrate the minimally required configuration, the following would approximate the current `service-pipeline-test-and-publish` JJB job for a project named "foo". pipelines: foo: directory: src/foo stages: - name: test # builds/runs "test" variant - name: candidate build: production publish: image: true deploy: # currently only the "ci" cluster chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true And to illustrate how the "candidate" stage in this example could be expressed as multiple stages using references to the output names that steps bind/export: pipelines: foo: directory: src/foo stages: - name: tested - name: built build: production - name: published publish: image: id: '${built.imageID}' exports: image: '${.imageFullName}:${.imageTag}' - name: staged deploy: image: '${published.image}' chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true Bug: T210267 Change-Id: I5a41d0d33ed7e9174db6178ab7921f5143296c75
5 years ago
pipeline: Builder and stage implementation Provides `PipelineBuilder` for reading `.pipeline/config.yaml` and mapping user-defined pipelines, stages, and execution graphs to actual Jenkins Pipeline stage definitions. Provides `Pipeline` class that constructs a "stack" of `PipelineStage` objects from the user-provided configs, each with its own `NodeContext` for binding output values to names and consuming bound values from previous stages. Provides `PipelineStage` that contains core stage step implementations based on the existing `service-pipeline` JJB job definition in `integration/config`. A closure is returned by each stage for passing off to Jenkins Pipeline stage definitions by the builder. Steps have a fixed order within a given stage: build, run, publish, deploy, exports. This allows for concise definition of a stage that performs multiple steps, and deterministic behavior of default configuration that references locally bound output values (e.g. the default configuration of `image:` for an `publish: { type: image }` publish entry is `${.imageID}`, referencing the image built in the current stage's `build` step.) If the user needs to change ordering, they can simply break the stage out into multiple stages. See the `Pipeline` class for currently supported configuration. Note that the aforementioned context system allows for user's to make use of the same value bindings that step implementations use internally. They can also use the `exports` configuration field to bind new values. To illustrate the minimally required configuration, the following would approximate the current `service-pipeline-test-and-publish` JJB job for a project named "foo". pipelines: foo: directory: src/foo stages: - name: test # builds/runs "test" variant - name: candidate build: production publish: image: true deploy: # currently only the "ci" cluster chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true And to illustrate how the "candidate" stage in this example could be expressed as multiple stages using references to the output names that steps bind/export: pipelines: foo: directory: src/foo stages: - name: tested - name: built build: production - name: published publish: image: id: '${built.imageID}' exports: image: '${.imageFullName}:${.imageTag}' - name: staged deploy: image: '${published.image}' chart: https://releases.wikimedia.org/charts/foo-0.0.1.tgz test: true Bug: T210267 Change-Id: I5a41d0d33ed7e9174db6178ab7921f5143296c75
5 years ago
  1. import groovy.mock.interceptor.MockFor
  2. import static groovy.test.GroovyAssert.*
  3. import groovy.util.GroovyTestCase
  4. import org.wikimedia.integration.Pipeline
  5. class PipelineTest extends GroovyTestCase {
  6. void testConstructor() {
  7. def pipeline = new Pipeline("foo", [
  8. blubberfile: "bar/blubber.yaml",
  9. directory: "src/foo",
  10. stages: [
  11. [name: "unit"],
  12. [name: "lint"],
  13. [name: "candidate"],
  14. [name: "production"],
  15. ],
  16. execution: [
  17. ["unit", "candidate", "production"],
  18. ["lint", "candidate", "production"],
  19. ],
  20. ])
  21. assert pipeline.blubberfile == "bar/blubber.yaml"
  22. assert pipeline.directory == "src/foo"
  23. assert pipeline.execution == [
  24. ["unit", "candidate", "production"],
  25. ["lint", "candidate", "production"],
  26. ]
  27. }
  28. void testConstructor_defaults() {
  29. def pipeline = new Pipeline("foo", [
  30. directory: "src/foo",
  31. stages: [
  32. [name: "unit"],
  33. [name: "lint"],
  34. [name: "candidate"],
  35. [name: "production"],
  36. ],
  37. ])
  38. assert pipeline.blubberfile == "foo/blubber.yaml"
  39. assert pipeline.execution == [
  40. ["unit", "lint", "candidate", "production"],
  41. ]
  42. }
  43. void testGetDefaultNodeLabels_build() {
  44. def pipeline = new Pipeline("foo", [
  45. stages: [
  46. [
  47. name: "foo",
  48. build: "foo",
  49. ],
  50. ],
  51. ])
  52. assert pipeline.getRequiredNodeLabels() == ["blubber"] as Set
  53. }
  54. void testGetDefaultNodeLabels_run() {
  55. def pipeline = new Pipeline("foo", [
  56. stages: [
  57. [
  58. name: "foo",
  59. run: [
  60. image: "foo",
  61. ],
  62. ],
  63. ],
  64. ])
  65. assert pipeline.getRequiredNodeLabels() == ["blubber"] as Set
  66. }
  67. void testGetDefaultNodeLabels_publishFiles() {
  68. def pipeline = new Pipeline("foo", [
  69. stages: [
  70. [
  71. name: "foo",
  72. publish: [
  73. files: [
  74. paths: ["foo/*"],
  75. ],
  76. ],
  77. ],
  78. ],
  79. ])
  80. assert pipeline.getRequiredNodeLabels() == ["blubber"] as Set
  81. }
  82. void testGetDefaultNodeLabels_publishImage() {
  83. def pipeline = new Pipeline("foo", [
  84. stages: [
  85. [
  86. name: "foo",
  87. publish: [
  88. image: [
  89. id: "foo",
  90. ],
  91. ],
  92. ],
  93. ],
  94. ])
  95. assert pipeline.getRequiredNodeLabels() == ["dockerPublish"] as Set
  96. }
  97. void testRunner() {
  98. def pipeline = new Pipeline("foo", [
  99. directory: "src/foo/",
  100. stages: [],
  101. ])
  102. def runner = pipeline.runner()
  103. assert runner.configPath == "../../.pipeline"
  104. assert runner.registry == "docker-registry.wikimedia.org"
  105. assert runner.registryInternal == "docker-registry.discovery.wmnet"
  106. }
  107. void testRunner_currentDirectory() {
  108. def pipeline = new Pipeline("foo", [
  109. directory: ".",
  110. stages: [],
  111. ])
  112. assert pipeline.runner().configPath == ".pipeline"
  113. }
  114. void testRunner_customRegistries() {
  115. def pipeline = new Pipeline("foo", [stages: []])
  116. pipeline.dockerRegistry = "registry.example"
  117. pipeline.dockerRegistryInternal = "internal.example"
  118. def runner = pipeline.runner()
  119. assert runner.registry == "registry.example"
  120. assert runner.registryInternal == "internal.example"
  121. }
  122. void testValidate_setupReserved() {
  123. def pipeline = new Pipeline("foo", [
  124. stages: [[name: "setup"]],
  125. ])
  126. def e = shouldFail(Pipeline.ValidationException) {
  127. pipeline.validate()
  128. }
  129. assert e.errors.size() == 1
  130. assert e.errors[0] == "setup is a reserved stage name"
  131. }
  132. void testStack() {
  133. def pipeline = new Pipeline("foo", [
  134. stages: [[name: "stageA"], [name: "stageB"], [name: "stageC"]],
  135. execution: [["stageA", "stageC"], ["stageB", "stageC"]],
  136. ])
  137. // Expecting:
  138. //
  139. // stageA
  140. // ⇗ ⇘
  141. // setup stageC ⇒ teardown
  142. // ⇘ ⇗
  143. // stageB
  144. //
  145. def stack = pipeline.stack()
  146. assert stack.size() == 4
  147. assert stack[0].size() == 1
  148. assert stack[0][0].name == "setup"
  149. assert stack[1].size() == 2
  150. assert stack[1][0].name == "stageA"
  151. assert stack[1][1].name == "stageB"
  152. assert stack[2].size() == 1
  153. assert stack[2][0].name == "stageC"
  154. assert stack[3].size() == 1
  155. assert stack[3][0].name == "teardown"
  156. }
  157. void testValidate_teardownReserved() {
  158. def pipeline = new Pipeline("foo", [
  159. stages: [[name: "teardown"]],
  160. ])
  161. def e = shouldFail(Pipeline.ValidationException) {
  162. pipeline.validate()
  163. }
  164. assert e.errors.size() == 1
  165. assert e.errors[0] == "teardown is a reserved stage name"
  166. }
  167. }