Simple unit tests for ensuring basic functionality of these utility
functions. Exported from the "config" package now for testing.
Change-Id: I5f7133d16c14f99ff0b3b6392e4f210357678a88
Adds a test to ensure that the new use-system-flag declaration is
handled correctly in a YAML config definition.
Change-Id: I551531e20f6b13b60314f4d1d86ac60c7aca9761
We attempt to install dependencies from pip to a custom location using
the --target flag. This flag is broken in Debian/Ubuntu due to a
distro-specific --user install default in versions prior to 18.1.1
(including the Stretch version).
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=830892
This behavior can be switched off by specifying the --system flag in the
install command. This patch adds support for a use-system-flag directive
that allows doing so.
Bug: T227919
Change-Id: Ifacc3d96be3a89135187e46450d524a7d083f885
We've changed the config to be more consistent about "copies" and
"artifacts". This means that "copies" no longer accepts a string, but a
slice of strings. We should bump the config version and blubber version
to make it clear that we're not longer backwards compatible.
Change-Id: Ib68f623cbdcdf0aed74f4fdaec062835748e8eb9
Update the go-playground validator to support "startswith" and
"endswith" validators to be used to validate images used for production.
See the example in policy.example.yaml for how this would be used.
Bug: T215319
Change-Id: Ie4362184bdb19399c667b4180b4bef32c2198063
Include a `.pipeline/config.yaml` with "test", "rehearse", and "publish"
pipelines defined which are scheduled by Zuul during "test",
"gate-and-submit", and "postmerge" respectively.
Depends-On: I835668eaee08bb4890cd1e3cfea2cdb84779352f
Change-Id: I3c63ad63b7d504a3204c6533a2cd8f00ca48492b
Work on Go Modules has been moving steadily with basic support in 1.11
and more mature support in the recently released 1.12. Eventually we
should look at using it for dependency management (away from `dep`) but
let's start by simply declaring the package name in a new `go.mod` to
allow building of our project outside the hell that is `GOPATH`.
Change-Id: I90b9f35194afe5538dce8db67b2b7c2d551bfaff
Add dependencies, gcc, git, and make, needed to build and test blubber.
These added depedencies were already present in the dockerhub
golang:1.10-stretch base image, but must be installed on top of the
leaner wmf golang:1.11.5-1 base image.
Change-Id: I75a3263b2e4635ed21a393531d901dcb9620453a
Refactored `copies` configuration to allow for greater control over when
and which both local build context files and variant artifacts are
copied into the target image.
The new configuration introduces a "local" keyword to signify when a
`copies` entry should apply to the files from the local build context as
opposed to files from another variant's image during a multi-stage
build.
variants:
build:
copies:
- from: local
source: ./src
destination: .
Note that with this change, the user must now explicitly define whether
_any_ files should be copied in from the local build context. None will
be copied in by default.
To help keep configurations succinct, especially considering this new
requirement, and to approximate the old `copies: variant`, a shorthand
format and sane defaults for `source` and `destination` (depending whether
`from` is "local" or a variant name) were implemented.
variants:
build:
copies: [local]
development:
copies:
- from: build
- from: local
source: ./config.dev.yaml
destination: ./config.yaml
The shorthand:
copies: [ref, ...]
# is equivalent to
copies: [{ from: ref }, ...]
And the following defaults are used when no `source` and `destination`
are specified.
copies:
- from: local
# defaults to
- from: local
source: .
destination: .
copies:
- from: variant
# defaults to two entries
- from: variant
source: /srv/app # the lives.in dir
destination: /srv/app
- from: variant
source: /opt/local # the shared lib dir
destination: /opt/local
Bug: T211625
Change-Id: I4c4217905afc0762b6bd66ed594d43cc0486e3e2
The paint order for the eye's outer stroke border is not respected by
the SVG preview renderer on commons. I re-implemented the eye's border
with two independent shapes to avoid the issue.
Change-Id: Ie450c4734072b9b215cbbb7e17b4648372610539
The new logo was the winner of a recent round of voting among Release
Engineering. (See https://phabricator.wikimedia.org/V19) It incorporates
colors from the WM design palette and the ring pattern from the logos of
other WM teams, platforms, and tooling.
R.I.P. Tyler's dead-whale logo.
Change-Id: I0f79dcd9cb521f39059fa4270f65508c3c097a54
Wrote an OpenAPI 3.0 spec for Blubberoid that provides `x-amples`
entries compatible with service-checker.
The written spec includes basic schema for Blubber config objects that
may be later factored out for use in validation.
Note that OpenAPI 3.0 supports only the v4 draft of the JSON Schema
standard, so some parts of the configuration could not be fully
described. Specifically, v4 does not include the `patternProperties`
definition introduced in the JSON Schema v6 draft that would allow us to
describe `variants` and `runs.environment` and everything beneath.
Blubberoid was refactored slightly to incorporate the new spec as well
as assume JSON as the canonical and default configuration format. It was
also refactored to include a versioned namespace ("v1") after the server
endpoint.
Bug: T205920
Change-Id: I28a341aa503b8920d802715660d4c4d62be45475
JSON seems a better option for a web service in general—other toolchains
in the Docker/Kubernetes space typically prefer YAML for human-edited
configs but convert to JSON on the wire. The "application/json" media
type is well established—unlike "application/yaml" which has no official
assignment by IANA—and is better supported by the OpenAPI (formerly
Swagger) specification.
Added content-type media type validation in the Blubberoid HTTP server
handler, and added a check for `json.Valid(body)` upon receiving a
"application/json" media type. Since any given valid JSON is also valid
YAML, Blubberoid simply does a shallow validation of the JSON body
before punting to `config.ReadConfig` for YAML unmarshalling and
thorough config validation.
Bug: T205920
Change-Id: I970acbde497ed446eb8eed568b1328f8c6f5aa55
Uses the github.com/ghodss/yaml library to convert YAML to JSON before
unmarshaling for the purposes of supporting YAML and JSON input while
converting to only support JSON internally.
Bug: T207694
Change-Id: I00668014907e9ea54917f5d5067cac08d0668053
Implemented additional rules in the Makefile to be more conventional.
Now you can do `make [all]` for both blubber and blubberoid binaries
without installing them, or `make blubber` and `make blubberoid` to
build them independently.
A `make clean` rule was also added.
Change-Id: Iaf52d9fbbc1c377c26a0cb21810af2e3d93778e5
Support for mounting a shared volume instead of copying application
files was meant to provide an option for development use cases. This
functionality has never been thoroughly tested or utilized for any use
case. It should be removed for now.
Relates tangentially to task T204591 that simplifies node support.
Change-Id: Ib70cb7bceb504841897a38d732880ba376fe67c8
The original design decision to install `node_modules` to a central
location outside the application directory was for a purported
development use case not yet realized. It has led to confusion and
unintended failures, and is just generally weird.
Let's right this wrong before it fossilizes into debt, like a hysterical
raisin in the sun.
Bug: T204591
Change-Id: I2e4797833471596b10b6b4a2705055050ca98ac6
Adding types to variables that would have their types inferred from
function return types causes lint errors in go1.10.3 (my local machine).
Change-Id: I71ffbe7843084b4bd22b15ed223ed05d4ccc9aaa
Missed during the rename from blubberd -> blubberoid. The shell needed
to be changed to /bin/bash to accommodate the "{}" shell expansion which
doesn't work with sh.
Change-Id: I1743d58b17fe794b517e8d1d947ae29135bdc73d
The `blubber` command already gets everything it needs from explicit
inputs, which makes it an easy candidate for running as a simple
microservice. This patch provides exactly that in the form of
`blubberoid`, an HTTP server that processes Blubber configuration.
To start the daemon:
make && blubberoid
To use it:
curl -i -X POST --data-binary @blubber.example.yamlhttp://:8748/[variant]
Change-Id: Ieea73048d092b974da424ba40ddc90eaf693af0b
The builder configuration has proven useful for supporting generic
pre-entrypoint commands such as dependency managers not otherwise
supported by specific Blubber configuration. Adding additional
`builder.requirements` config expands support for such commands by
allowing the user to specify files that should be copied into the image
before the builder command runs.
To support this extra configuration, `builder` had to be changed from a
simple string to a mapping. The builder command must now by given as
`builder.command`.
The pattern of creating parent directories, copying files, and executing
one or more commands prior to the entrypoint has become a common
one. Some of the implementation of this pattern was moved from
`PythonConfig` into shared build macros `build.SortFilesByDir` and
`build.SyncFiles`. All config types that must have requirements files
copied over independently of the entire source tree (`PythonConfig`,
`BuilderConfig`, `NodeConfig`) now delegate to these functions.
Change-Id: I67f33034f22cee2851ec866cfb07ab20c23eba8c
Added a `blubber.yaml` that defines a single test variant and entry
point for use with the new generic `blubber-test` CI job (see
I31b367f2ab8b897fa988e70222f70d84186a631b).
Bug: T200452
Change-Id: I60b23e1bca8486e43ebf0f62198e85833f86b921
The `lint` rule was previous surfacing purely whitespace output of
`gofmt` as a failure and running `go tool vet` instead of `go vet` (the
former accepts source files as arguments, the latter takes package
names).
A real lint error in `config/node.go` was also fixed.
Change-Id: I7f13dcbb45a448f1ecac87e97527a7f4ed4d2c48
With the move to Gerrit and away from Phabricator/Arcanist, we lost an
easy way to run linters. New rules have been added to the `Makefile`,
`lint`, `unit`, and `test` that run linters, unit tests, and both
linters and unit tests, respectively.
Bug: T200452
Change-Id: I0742daaa14389841d88f13eba47dee07ac127cf2
All import paths and other references to the previously Blubber repo in
Phabricator have been changed to point to the new Gerrit project URL.
Note that building or testing against this change will require you to
move your working directory to `$GOPATH/src/gerrit.wikimedia.org/r/blubber`.
Contribution documentation was updated to reflect the changes, and the
`.arcvendor` submodule has been removed.
Bug: T200452
Change-Id: I3ab23e420b2825e86e2bd7c9b3da9d4de23edaa1
Summary:
Ran into a Docker error when attempting to build a node project with
more than one requirements files:
When using COPY with more than one source file, the destination must
be a directory and end with a /
Reviewers: dduvall, #release-engineering-team
Reviewed By: dduvall, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D1080
Summary: Bump version to v0.4.0.
Test Plan: Run `make && blubber --version` and verify the output.
Reviewers: thcipriani, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D1060
Summary: See upstream bug https://github.com/golang/go/issues/24359
Test Plan:
Run `make` from a symlink targetting the blubber directory under `GOPATH` and
verify that `blubber --version` outputs the version and commit.
Reviewers: thcipriani, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D1061
Summary:
Change `node.dependencies` flag to `node.requirements`, the list of
files (typically `package.json` and either `npm-shrinkwrap.json` or
`package-lock.json`) that declare dependencies to be installed with NPM.
Test Plan: Run `go test ./...`. Try it out with something like Mathoid.
Reviewers: thcipriani, hashar, mobrovac, Joe, akosiaris, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D1058
Tyler Cipriani
jenkins-bot
Michael Holloway
Dan Duvall
Lars Wirzenius
Antoine Musso
Gilles Dubuc