Simple unit tests for ensuring basic functionality of these utility
functions. Exported from the "config" package now for testing.
Change-Id: I5f7133d16c14f99ff0b3b6392e4f210357678a88
Adds a test to ensure that the new use-system-flag declaration is
handled correctly in a YAML config definition.
Change-Id: I551531e20f6b13b60314f4d1d86ac60c7aca9761
We attempt to install dependencies from pip to a custom location using
the --target flag. This flag is broken in Debian/Ubuntu due to a
distro-specific --user install default in versions prior to 18.1.1
(including the Stretch version).
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=830892
This behavior can be switched off by specifying the --system flag in the
install command. This patch adds support for a use-system-flag directive
that allows doing so.
Bug: T227919
Change-Id: Ifacc3d96be3a89135187e46450d524a7d083f885
We've changed the config to be more consistent about "copies" and
"artifacts". This means that "copies" no longer accepts a string, but a
slice of strings. We should bump the config version and blubber version
to make it clear that we're not longer backwards compatible.
Change-Id: Ib68f623cbdcdf0aed74f4fdaec062835748e8eb9
Uses the github.com/ghodss/yaml library to convert YAML to JSON before
unmarshaling for the purposes of supporting YAML and JSON input while
converting to only support JSON internally.
Bug: T207694
Change-Id: I00668014907e9ea54917f5d5067cac08d0668053
The builder configuration has proven useful for supporting generic
pre-entrypoint commands such as dependency managers not otherwise
supported by specific Blubber configuration. Adding additional
`builder.requirements` config expands support for such commands by
allowing the user to specify files that should be copied into the image
before the builder command runs.
To support this extra configuration, `builder` had to be changed from a
simple string to a mapping. The builder command must now by given as
`builder.command`.
The pattern of creating parent directories, copying files, and executing
one or more commands prior to the entrypoint has become a common
one. Some of the implementation of this pattern was moved from
`PythonConfig` into shared build macros `build.SortFilesByDir` and
`build.SyncFiles`. All config types that must have requirements files
copied over independently of the entire source tree (`PythonConfig`,
`BuilderConfig`, `NodeConfig`) now delegate to these functions.
Change-Id: I67f33034f22cee2851ec866cfb07ab20c23eba8c
All import paths and other references to the previously Blubber repo in
Phabricator have been changed to point to the new Gerrit project URL.
Note that building or testing against this change will require you to
move your working directory to `$GOPATH/src/gerrit.wikimedia.org/r/blubber`.
Contribution documentation was updated to reflect the changes, and the
`.arcvendor` submodule has been removed.
Bug: T200452
Change-Id: I3ab23e420b2825e86e2bd7c9b3da9d4de23edaa1
Summary:
We want to also install the wheels, not just generate them. To do that
we add a pip install -t <target> command. For this to work however, we
need to install into /opt/lib/python and amend the environment
accordingly to set PYTHONPATH and PATH in order for the libraries and
binaries to be accessible.
Reviewers: dduvall, thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D1057
Summary:
Since `config.Validate` was changed to take any interface as an
argument, many of the validation tests can be refactored in a way that
avoids having to parse the full config context in YAML and instead
validates each specific config struct directly. The new test pattern is
simpler and less prone to future breakage should unrelated parts of the
overall config change.
Tests that rely on root config context were left unchanged.
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, demon, hashar, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D1023
Summary:
Introduced a `version` config field that must be specified and match
`config.CurrentVersion`.
Changed `config.ReadConfig` to use `yaml.UnmarshalStrict` to ensure that
errors are surfaced when unknown/bad fields are present in the given
YAML config. A smaller `config.VersionConfig` is now unmarshaled first
to prevalidate the new `version` field before the entire config is
parsed.
Fixes T191460
Test Plan:
Run `go test ./...`. Run `blubber` against some configuration containing
invalid fields and ensure that it surfaces a YAML error.
Reviewers: thcipriani, demon, hashar, mmodell, mobrovac, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T191460
Differential Revision: https://phabricator.wikimedia.org/D1021
Summary:
A new root and variant `python` config field is provided with two new
fields below, `version` and `requirements`.
The former, `version`, should specify the Python executable to use when
executing related package installation commands and ostensibly the same
executable that will be used to run the application.
The latter, `requirements`, should specify all pip requirements files
such that a compiler that supports layered filesystems (e.g. Docker) can
output separate instructions that will invalidate cache layers for
changes to those files independently of changes to the rest of the
codebase.
Python related instructions will be generated only if either `version`
or `requirements` are given.
Fixes T186545
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, hashar, demon, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T186545
Differential Revision: https://phabricator.wikimedia.org/D976
Tyler Cipriani
Michael Holloway
Dan Duvall
Alexandros Kosiaris